January 31 Thursday

• CIPT (5-11:30pm) - [Jeff Kern]
  • 23:15MTS After Peter's work, Jeff, Rafael, Robert, and Marcus worked on ASDM verification and CORR software until 23:00MTS. After that science wasn't be done. telescopes in park position and locally locked.
• PSI (1-5pm) - [Peter Napier]
  • A report from Peter (extracted by Debra based on e-mail):
    • Yesterday we adjusted the timing in a couple of the ATF digitizers and succeeded in removing the spurious 250 MHz harmonic spurs that Sci has seen in cross-correlation spectra. Digitizers in BBPr0-PolXX in AEC and BBPr1-PolXX in VA were adjusted. I will issue a separate report on this.
    • For this work we looked at all 4 currently available correlator outputs and they all now look usable. We effectively have two independently tunable basebands that are available simultaneously out of the correlator. If it would help with the baseline determination work we can tune these basebands to opposite ends of the 4-12 GHz IF and do simultaneous baseline observations with frequencies separated by 8 GHz.
    • The 4 outputs from the correlator are listed below and can be selected on CorrGUI. Polarizion xx OR yy is available in single polarization mode in CorrGUI. Polarization xx AND yy are available simultaneously if you select 2 Polarization mode in CorrGUI. But note that in the current configuration x and y both come from the same single FE polarization.
    • Currently available correlator outputs:
      • Baseband Pair 0, Polzn XX (BBPr0-XX). This is the baseband currently used for astronomy. It's frequency is controlled by LO2 0x40.
      • BBPr0-YY, Frequency controlled by LO2 0x40. This is essentially an identical piece of spectrum to BBPr0-XX but it is delivered through a completely separate baseband and digital channel.
      • BBPr1-XX, Frequency controlled by LO2 0x41. Available to tune to a different frequency to BBPr0.
      • BBPr1-YY, Frequency controlled by LO2 0x41. This is essentially an identical piece of spectrum to BBPr1-XX but it is delivered through a completely separate baseband and digital channel.
  • 250 MHz spurious harmonics: BE personal were able to remove the spurious signals seen in the spectrum by adjusting the clock on the digitizer (DTX). So this problem was due to a timing error. Now the cross correlated image only show the expected peaks comming from Intermediate LO. Hector
• CIPT (8am-1pm) - [Robert Lucas]
  • Late start due to power down of PSA overnight (1.5 Hr.). Unfortunatly the archives were not run so the cause is unknow.
  • Found bug in CCL which was preventing ABM Containers to fail. Fixed but out of time...will work with Science this evening to continue effort.

• PSI/VA: Gene reported the VA PSA outputs are off. Confirmed and contacted Jason in BE and he is trying to troubleshoot but software is not cooperating. Jeff Kern is working on the software. To be continued........ Jack
• PSI/VA: AIV needs the VA OPT power cable that is installed inside of a conduit on the VA antenna. After being severely faught, I managed to get the cable out of the conduit. Now the harder part, ship the cable ASAP to AIV in Chile. - (2-hours) Jack
• BE/AEC/VA: Paula and Eric (assisted by Peter) are here to troubleshoot 250mhz spike issues in the correlated data. Good results, no more 250mhz spikes!
• PSI/AEC: Instructed the two nubee SCI types on how to lockout the replacement AEC antenna lockout switch. - (.25-hours) Jack

• Activity Log: - Active: 15.5 (development), Passive: 0.5 hrs, Unscheduled: , Downtime: 8 hrs (PSI archives were not run).

January 30 Wednesday

• SCI (17:00 - 22:00)
  • atf.science.obs.31jan08.doc: atf science obs summary for 30jan08
  • 22:00MTS Operator is leaving ATF. System is tracking and getting fringes smoothly. Hector
  • 17:00MTS Science started on time. Weather conditions are good enough for observations. Now we are checking focus. It was set at -500 for VA and -4700 for AEC. Hector.

• DTX (14:00 - 17:00) - PSI/AIV (09:00 - 17:00)
  • 16:30MTS PSI could not connect to AEC ACU. It said that is "unavailable". After a while the error disapeared without do anything.
  • DTX 15:30MTS Test finished. Some problems with CORR subsystem after it. Seems to be confused. Jeff was called and suggested to shutdown everything and start up again.
  • DTX 14:25MTS. Paula started her DTX test.
  • 14:00MTS Jeff is working on to avoid software problems after remove optical telescope. DTX is not working yet because of that. Hector
  • 12:30MTS TELCAL and EXEC containers appear down because of the optical telescope was removed on AEC. Ralph was called and a software intervenvion should be done to take this in account.
  • Weather condition are bad for observations. Is snowing and windy. Antennas in SurvivalStow position. Hector.

• CIPT (08:00 - 13:00) LO sw testing
  • reconfigured to use my INTROOT. This included
    • Changing developeEnv
    • restarting daemons
  • Built new code into my INTROOT. Besides the modules I wanted to test (Array and AntLOController^?) this included the TMCDB and Mount component (as the correct versions of these were in the INTROOT I replaced)
  • Disabled AMBManager access to prevent unexpected commanding of the equipment I was testing and incorrect monitoring cluttering the logs (and hiding the information I was lookoing for)
  • Found I could not communicate with any the equipment in the BE analog rack on the Vertex antenna. Eventually solved by Jack by power cycling the rack. Lost about 90mins because of this.
  • looked into why the LO2 component was reporting changing frequencies. It was because the relevant monitor point is actually changing as a result of the automatic fine tuning. Discussed this with Sylas and Jeff. This will be fixed in Fridays firmware upgrade. Correct solution for obtaining the LO2 frequency is to trim this value to the nearest comb line and then add the FTS frequency. The LO2 component was changed, by Jeff, to do this.
  • Looked into why the LO2's would not stay in lock. Problem appears to be the FTS frequency is unexpectedly being set to zero by the LO2 component. Jeff and Ralph are looking into this. Ralph could not reproduce the result on the simulator.
  • private INTROOT removed, system shutdown, daemons restarted and CDB modified to reenable AMBManager access.
  • At 1:20 handed the system over to Jeff and Nicholas Troncoso to reconfigure for the removal of the optical Telecsope.

• PSI/AEC: Removing OPT-2, power supply, cables, fibers, sealing holes. Done. (4-hours) Jack & Juan G.
• PSI/VA: Ralph reported he couldn't communicate with the analog rack, power cycled the rack and now he can communicate. (.5-hours) Jack
• PSI/AEC: Continued to install the replacement antenna lockout switch. Done. (2-hours) Jack
• PSI/CLO: Set up 3 temperature data loggers in the CLO, Aux. CLO, and room temp. Data for Gene and Bill Shillue. (.5-hours) Jack

• Activity Log: - Active: 7.5 hrs (development) + SCI(7), Passive: 0 hrs, Unscheduled: , Downtime 8 (PSI scripts not run for 8 hrs) + 1.5hrs (against development - BE analog racks)

January 29 Tuesday

• SCI: (17:00 - 00:00) Robert&Silvia + CIPT Marcus overlap
  • atf.science.obs.30jan08.doc: atf science obs summary for 29jan08
  • 00:00MTS Operator is leaving ATF. Everything is working fine. Hector.
  • 17:00MTS System is back and science started checking the focus position. Subreflector control from AEC mountPanel. New focus vales are -500 for VA and -5700 for AEC. Hector.

• PSI: (15:00 - 17:00) Jack replace lock-out switch
  • 16:00MTS. System didn't came back normaly. Nicolas had a look on this problem and noticed that ACS didn't went down. He had to kill ACS manually. Hector
  • 15:00 MTS. After Robert job, tried to move antennas to survival stow position sending the command from mountPanel. The antennas moved randomly and seemed that panels were very confused. Something to note is that Robert left an array created and probably that is related with the problem. Jeff was called to have a look on this and suggested to do power cycle on both ABMs and restart everithing. For the moment DO NOT USE SURVIVAL STOW COMMAND FROM MOUNTPANEL. Hector

• CIPT: (08:00 - 15:00) Robert Lucas - ASDM Verificator
  • 15:00MST ASDM Testing stopped. Started late (~11:00) due to Vertex problems. Could n't make it work this time, testing will resume on Thursday (Robert Lucas)

• • PSI/ATF: Checking e-mails, google calendar, writting at Journal. (Juan 2.5 h)

• • PSI/ATF: Reading about cryogenic (Juan 3 h)

• • 11:11MST All Failures cleared, the Vertex antenna operates normally. (Juan 1.5 h , Nicolas)

• • 10:14MST Software is operational. The mountPanel for vertex reports a problem with the PTC computer. Juan will reboot the PTC(Juan 0.5 h).

• • 09:40MST Antenna VA is in limit. Juan just arrived is fixixng that. CIPT will use down time to reboot gns, and change UPS battery.

• PSI/AEC: Wired and installed replacement lockout switch, will connect tomorrow. (1.5 hours) Jack, Juan G.
• PSI/AEC: Looked over antenna wiring and found a breakpoint for the lockout switch installation so as not to kill entire antenna power tomorrow. (.5-hours) Jack, Juan G.
• PSI/AEC: Went over OPT-2 removal procedure with Juan. (.5-hours) Jack
• 14:20MTS Put a sandbag on the Halogen light support to avoid that the wind would take it away. Juan
• 09:35MST PSI/ATF: Cryo temps and pressures status at 9:35 AM: VA=160 PSI on Tank, AEC=90 PSI on Tank. (Juan 0.25h)

• Activity Log: - Active: Development(9)+SCI(7), Passive 0 h: , Downtime 8 hrs (PSI archives not run)

January 28 Monday

• SCI: (17:00 - 00.00) Science time in parallel with CIPT Marcus
  • atf.science.obs.29jan08.doc: atf science obs summary for 28jan08
  • 00:15 - Antennas locked out; Alcatel in survival stow; Vertex in limit position.
  • 23:16 - Vertex runaway to elevation + hard limit. Cannot recover from software.
  • 17:28 - 23:16 On 3C84 with fringe tracking on LO1 and LO2.

• 17:50 -- PSI/ATF: Cryo temps and pressures status at 17:50 PM: VA=172 PSI on Tank, AEC=91 PSI on Tank. (Juan 0.25h)

• AOPG: (17:00- 18:00) Position Stream measurement to be done by the Alma Operation Group (Hector and Emilio)
  • 17:00 -- Done in the morning with no runaway. Data is under ~halarcon . Time given to SCI. [ebarrios]

• CIPT (13:00 - 17:00) ASDM and SFI testing
  • Established fringes using CorrGUI as a testbed for an Array formed by the online system
  • Demonstrated fringes with fringe tracking in both first and second LOs. Some issue with interaction between PSI scripts and the online software, not clearly understood but we were able to work around it.

• PSI/ATF: Watching the operator and Science work at the main trailer. (Juan 1.5 h)

• PSI: (09:00 - 13:00) System Test by Peter Napier - no down time. Repeated Walsh Function tests of 22 Jan using correct Walsh Function psi scripts supplied by Gene DuVall. Main conclusions: In TDM mode Walsh Function phase switching correctly removes spurious signals in cross-correlation including a large DC offset in Channel 0 of the spectrum; in TFB mode Walsh Function phase switching correctly removes spurious signals in cross-correlation although there is a puzzle in that there is no DC offset in Channel 0 to be removed; confirmed again that Walsh function switching removes the 250 MHz spurious signals that have appeared in the cross correlation indicating that these spurious signals are generated by the system ahead of the DTX formatter; for the first time used CorrGUI to look at BaseBand^? Pair BBP) 1 (rather than the usual BBP 0) and found that the spurious 250 MHz signals are much stronger in BBP 1 than BBP 0.
  • 13:00 -- Jeff start interferometry new setting and test. [ebarrios]
  • 10:00 -- Peter started his test. Hector

• • PSI/ATF: Checking e-mails, google calendar, add info to Journal. (Juan 2 h)

• 09:55 -- PSI/ATF: Cryo temps and pressures status at 9:55 AM: VA=174 PSI on Tank, AEC=94 PSI on Tank. (Juan 0.25h)
• 09:15 -- We start the PositionStreamClient^? recording on Vertex around the run away position. [ebarrios]
• 08:45 -- ARCHIVE subsystem was found in ERROR status. We shutted down and started up back all the system. Hector
• ??:?? -- PSI/ATF: Software down, no system check possible. - Jack

• Activity Log: - Active: PSI(4)+CIPT(4)+SCI(6) = 14h, Unscheduled: , Downtime

January 27 Sunday

• SCI: (14:00 - 23:45) Robert & Itziar
  • atf.science.obs.28jan08.doc: atf science obs summary for 27jan08
  • 23:20 -- Conditions slightly better. Tried to observe; DA41 container crashed and Alcatel wouldn't move. Restarted container; dis/reconnect mount Panel; control -> shutdown -> operational. Alcatel now under control but Vertex is not. Wind speed close to limit so moved Alcatel to survival stow and locked out. RAL
  • 20:30 - 23:20 -- Antennas at survival stow. Conditions too poor to observe RAL
  • 20:24 -- Vertex runaway, this time to low elevation. Recovered. RAL
  • 20:00 - 20:20 -- Pointing on Mars. Observing conditions very poor; cloud and high wind. RAL
  • 20:00 -- Operator leave the ATF. [ebarrios]
  • 18:45 -- After following the system startup procedure the chopper test shows problem with ALCATEL signal. After some rs sus, rs csib104l commands we call Jack. His suggestion was to plug OUT/IN the Band3 MUX module at the front-end rack. I manage to do it and after a second try we got better signal. To be checked tomorrow with Peter. [ebarrios]
  • 17:00 -- Not sure on the time but Marcus arrive and we manage to show him how fast/slow could be the shuttdown and startup procedure from the OMC panel. Also some comments on the subject were given to him. [ebarrios]
  • 15:00 -- We got a point where it was not possible to lock the wca2. Juan's advice was to check the AEC optical power which was around 0.16 instead 0.25. So the optical power was adjusted by Juan to ~0.24 but it didn't fix the wca2 lock. A workaround to fix this problem was found and written down by Robert. See his report. [ebarrios] (Juan 1h).
  • 14:00 -- Robert & Silvia start a troubleshooting trying to figure out why WCAs couldn't be lock after crash while the fringe tracking was on. [ebarrios]

• PSI/ATF: -- Cryo temps and pressures status at 5:00 PM: VA=180 PSI on Tank, AEC=97 PSI on Tank. (Juan 0.25h)

• CIPT: (13:00 - 16:00) Marcus
  • 13:30 -- System was re-started and lock ready for interferometry. As Marcus couldn't reach ATF on time Robert & Silvia took the control. [ebarrios]
  • 13:00 -- As it was suggested we (Juan and Me) check the Alcatel Elevation noise and brake. Noise just disappear and this time by manualy releasing the elevation brake the antenna moves smoothly which was not the case the previous day. [ebarrios] (Juan (0.5 h)

• PSI/ATF: Reading documentation of prototype antennas. (Juan 2 h)

• PSI/ATF: Checking e-mails, google calendar, Journal. (Juan 2 h)

• CIPT: (09:00 - 13:00) Jorge & Pablo Regresion Test
  • PSI/ATF: Cryo temps and pressures status at 9:40 AM: VA=177 PSI on Tank, AEC=96 PSI on Tank. (Juan 0.25h)
  • 09:00 -- Starting Regresion Test

• Activity Log: - Active: CIPT(5)+SCI(5), Passive: 9h, Unscheduled: 0h, Downtime: 2h on CIPT time (due to HW) 1h on Sci time (due to SW), 2h on Sci time (due to bad weather)

January 26 Saturday

• SCI : 16:00 - 00:00 Robert & Silvia
  • atf.science.obs.27jan08.doc: atf science obs summary for 26jan08
  • 22:00 -- Back on operation but now doing pointing sequence. [ebarrios]
  • 21:00 -- While in track we lost the the Alcatel WCA2 lock and fixed again by stopping delay server+objexp. HW Downtime (1h). [ebarrios]
  • 20:00 -- Back on source but still with low signal. Alcatel focus value was 0 and trying -5000um signal improve then a focus sequence was done starting from -10000 to 0 with step size of 2500um. Best on -5000um. [ebarrios]
  • 18:30 -- After full successfully setup we were on source getting fringes, but the expected signal was low. Guessing a possible problem with the Alcatel elevation encoder axis initiazation we try to point other source. At this point Elevation problem happen again. I couldn't move it with the PCU+handset control. So I repeat the Alcatel ACU power cycle to get the elevation control back. After that we call Jack to confirm the encoder initialization command, i.e "rs move aec ini". After recovering the mountPanel shows a complete wrong elevation position, but the corresponding intialization fix it. Finally back on operation with the advice to Juan Gallardo to check the elevation noise which to my experience it's like friction and accordingly to Jack it should be very quiet. After that we couldn't lock the wca2. Drew was call but after some measurements no success. Some how the locking phase problem was fixed by stopping delay server + objexp on crc-02. Back in operation around 20:00. HW Downtime (1.5h). [ebarrios]
  • 18:00 -- Due to problem with the Alcatel elevation axis and slow Acs re-start SCI start with a HW 2h of downtime. [ebarrios]

• CIPT: 09:00 - 16:00 Regression test (Jorge & Pablo)

• PSI/ATF/CIPT: During the regresion test once Jorge press the stop button on the mountPanel, a failure appeared on AEC Antenna (Drive motor #2 Failure in Elevation Axis). We thought that it could be something related to the brakes we checked the oil level and refill it but it wasn't that the cause. We called Jack and he direct us to powercycle the ACU using the swicht in the back of it. This action fixed the problem (Emilio, Juan, Jorge, Pablo, Jack, 2.5h).
• PSI/ATF: Giving support to people to solve a failure appeared on AEC Antenna (Drive motor #2 Failure in Elevation Axis. This means use the manlift to check brakes on AEC, use PCU, ACU (Juan 2.5 h)
• PSI/ATF: Checking e-mails, google calendar, Journal. (Juan 1 h)
• PSI/ATF/: UPS next to the Snickers PC is again asking for battery replace. 3 PC are connected to it plus one lamp. A new outlet was added to connect the lamp direct to the power instead of the UPS. (Juan 0.5h)
• PSI/ATF: Cryo temps and pressures status at 5:30 PM: VA=182 PSI on Tank, AEC=96 PSI on Tank. (Juan 0.25h)
• PSI/ATF: Cryo temps and pressures status at 9:40 AM: VA=170 PSI on Tank, AEC=86 PSI on Tank. (Juan 0.25h)

• Activity Log: - Active: CIPT(5.5)+SCI(3.5) = 9h, Passive: 9h, Downtime: CIPT(1.5)+ SCI(4.5) all due to HW = 6h

January 25 Friday

• Sci:
  • atf.science.obs.26jan08.doc: atf science obs summary for 25jan08

• PSI : 17:00 - 00:00 No crashes, but, some cosmetic issues.[ebarrios]
  -- Around UT02:34 the OMC panel subsystems change their state from operational to Err but every thing was running OK. After few seconds it change back to operational (panel refreshing ??). [ebarrios]
  -- Once on source from time to time for a few seconds on both mount panels - but not syncronized - the state "In position" change to NO as well as the commanded and deviation status line change to blank. Again no real problem (panel refreshing ??). [ebarrios]
  -- UPS next to the Snickers PC is asking for battery replace. [ebarrios]
  -- PositionStreamClient^? run twice on Vertex and one on Alcatel. First measurement on Vertex shows problems. Corresponding png files are under ~ebarrios/data .[ebarrios]

• CIPT : 08:00 - 17:00 Installed 13 December build of CASA on gas, gns, crc-01 & crc-02. Installed rpms needed by CASA on crc-01 & crc-02, but saw missing dependencies on gas and gns. CASA came up without errors on crc-01, but gave warnings about missing readline on crc-02. E-mailed ITS about the need for a proper installation. Debugged DV01 container crash on shutdown, and implemented a possible fix. Attempted to run SFI all the way through but couldn't Archive ASDM because of Archive schema mismatch. Archive schemas reloaded and successfully archived and retrieved 2 TDM (256 channels) asdms but had some problems with spectral window data in TFB (8k channels) mode. Due to mostly human error and OMC freezes we ran out of time to do this with fringes. Noted problems with excessive logging to Archive ACC/javaContainer and excessive CPU activity of jDAL after crash of OMC. (Joe S., Jeff, J. Perez, Rafael, Robert L.)
• PSI/ATF: Talking with operators about ALMA issues and tasks. (Juan 1 h)
• PSI/ATF: Reading documentation of the project, AIV reports, etc. (Juan 3 h)
• PSI/ATF: Cryo temps and pressures status at 4:30 PM: VA=184 PSI on Tank, AEC=97 PSI on Tank. (Juan 0.25h)
• PSI/ATF: Cryo temps and pressures status at 10:00 AM: VA=168 PSI on Tank, AEC=89 PSI on Tank. (Juan 0.25h)
• PSI/ATF: Checking e-mails, google calendar, Journal. (Juan 2 h)

• Activity Log: - Active: CIPT(10)+Sci(7)= 17hrs, Passive: 7 hrs, Downtime: 0.0hrs

January 24 Thursday

• Sci
  • atf.science.obs.25jan08.doc: atf science obs summary for 24jan08

• PSI/ATF: Cryo temps and pressures status at 5:00 PM: VA=179 PSI on Tank, AEC=89 PSI on Tank. (Juan 0.25h)
• PSI: 17:00 - 00:00 Robert and Silvia. After the runaway troubleshooting, system setup was done from scrash. To change the login frequency Robert&Jeff change the logLevel of some container with a side effect which force a new Acs down/up. After a while on source no data was stored because the disk was full. Jorge Sepulveda was called to do some cleanup, but, in parallel we manage to keep going with the measurements by changing a symbolic link to use the data3 disk. Every thing was smooth till "corr" subsystem when to "err" and as the correlator was still working, i.e. measurement on corrGUI were OK, we just keep going. Finally around midnight corrGUI complain because time out and one of the container was down. We were running the second part of the last sequence so we decide to close the night and leave the system in the failure state for CIPT. [ebarrios]

           Details given by raviles 
           21:00MST, SCI Laing, Leurini and Lucas are doing pointing tests offseting the antenna ~ 52.5"
           around the nominal position on which the system is already getting fringes on 3C84. The night
           had been cloudy to North-West. Every times the antenna offset in (+-)AZ/EL they take a note of
           the power cross correlation level fluctuation (in db.) This work was interrupted since the /data2 
           partition was full, which later was fixed by CIPT Jorge Sepulveda. The system failed a number of 
           times at the beginning of operation and the way to fix that was changing the logging level on 
           Control/{ant} cppContainer , MountController and Trajectory Planner Thread. Through the "failures", 
           the  status of VA and AEC mount panel changed from AUTONOMOUS to 'Delay Getting State' and then back 
           to AUTONOMOUS and repeated a number of times. This effect was more notoriuos on VA than in AEC. [Roberto]
       

• AOPG: 15:00 - 17:00 ALMA Operation Group try PositionStreamClient^? script on both antenas around the position were a runaway was observed. After some syntax error clarified by Jeff, we manage to reproduce the runaway on Vertex but the script was not running. The following are an entries done by Roberto. [ebarrios]

           PositionStreamClient Syntax
           The online help of the script states the way to use it is:
           PositionStreamClient {ant_name} {output_file}> , which is wrong, 
           is needed to redirect the output using ">"
  
           17:00MST, a number of tests with Position Stream Client(hereafter PSC) had been ran on VA, 
           while trying to reproduce the failure observed yesterday. The PSC hungs at least once in the
           same way it was reported in the JIRA ticket COMP-1680. A number of times, the cppContainer of 
           both dv01 and da41 failed (red flag.) It was observed the maci container was still running on 
           both machines. Apparently after 10 minutes the maci process finally dissapeared by itself from 
           DV01. A short test was done on AEC using the steps given by CIPT last Monday:
           dv01-abm>
           gdb attach 
           gdb> set logging file  (create a file and give this name to CIPT, 
                                             particularly to Nicolas Troncoso and Nicolas Barriga.) 
           gdb> set logging on 
           gdb> bt full (backtrace) 
           gdb> info threads (thread info) 
           gdb> thread apply all backtrace full (backtraces for all threads, takes some time).
           The name of the output file was given to  Nicolas Barriga and Nicolas Troncoso (through email.) 
           After the test the maci container process had dissapeared, we do not know if during or after the test. 
           [Roberto]
       

• BE/CORR: 13:00 - 15:00 paula will NOT be visiting the ATF today to redo the firmware of the DRX's. she does still need the time for remote access for M&C. there could be some new issue, but this time would help resolve this; there is some new insight she found with possibly making it repeatable on the bench. Done w/o downtime

• PSI/ATF: Reading documentation of the project, AIV reports, etc. (Juan 3 h)

• CIPT : 10:30-5:150 Tested the gdbBTPull.py script to see whether it would allow a fully loaded ABM container to continue running -- it wouldn't. Started the container using valgrind and sent the output to Bogdan & Gianluca; we need to find out why it doesn't work on a real-time kernel. Examined three-minute gap in container logs reported by Ralph from a run that he did yesterday morning; after checking with Paula that it was OK to do, we tried to run Ralph's test (with help from Ralph) but didn't succeed. Turned the system over to the operators. (Joe S. & Gianluca)

• PSI/ATF/BE:Silver Sturgis and Michael Pursley are in need of pictures of the front of the digital and analog racks with the doors open in the AEC antenna. The pictures should clearly showed how the the modules in each rack are laid out. These pictures are for future reference for the High Alt simulation made on December second.(Juan 0.5h)
• PSI/ATF: Checking e-mails, google calendar, Journal. (Juan 2 h)
• PSI/ATF: Cryo temps and pressures status at 10:00 AM: VA=172 PSI on Tank, AEC=90 PSI on Tank. (Juan 0.25h)

• Activity Log: - Active: 9 hrs (development) 7 hrs (sci), Passive: 8 hrs, Downtime: hrs

January 23 Wednesday

• Sci:
  • atf.science.obs.24jan08.doc: atf science obs summary for 23jan08

• PSI/ATF: Cryo temps and pressures status at 4:00 PM: VA=182 PSI on Tank, AEC=96 PSI on Tank. (Juan 0.25h)
• SCI : 20:00 - 00:00 Science by Robert (downtime 20:30-23:00 = 2.5h). After about 1/2h on 0359+509 (03:59:29.74 ; 50:57:50.161 ; ST 05:00:00) a Vertex runaway in Elevation got the harware limit around 125Deg. It was manually moved out of the limits and then to the stow position using the ACU touch panel console. We try to repeat the failure and it happen again (AZ ~-24 El~71), but after that no other ocurrency. After some confusion trying to figure out why Vertex signal was low, we back on sky. Vertex signal still low. [ebarrios]
• On above entry: Same failure was observed and reported (email) by Manuel Olivares and Hector Alarcon last December 8th, 2007; see edited summary below and note coordinates are not the same of the above reported event:

VA went to hard limit (125 degrees) when (using the mount panel) we tried to move to a new position:
Start position (3c446) : Ra 22:25:47.26   Dec: -04:57:01.39
New position (3c454.3): Ra 22:53:57.75  Dec : 16:08:53.56
This happened at 03:05 UT or 08:05 Local time.
...system...error message:  Error Executing command 5 ID=5
Current Build ALMA 4_1_3_1

Antenna was recovered using the VA-ACU touch screen.[Roberto]

• CORR : 17:00 - 18:00 Firmware to 5.0.1 not done. Time used by CIPT (not downtime). [ebarrios]
• CIPT : 08:00 - 13:00 LO chain tuning on 5.0.2 . Switching back to 5.0.1 not done. [ebarrios]
• report on LO tuning tests RalphMarson
  • Not possible to start at 8:00am as there was a problem with the previous nights switch from 5.0.1 to 5.0.2. CIPT support staff looked into it and concluded that the problem was that the software switch was done on the acc and not on the gns. This problem will go away when COMP-2093 is resolved. * Started with the system at 10:00am. Immediately shutdown and
    • disabled AMBManager access to prevent unscheduled access to the hardware I was manipulating.
    • restarted the daemons to ensure they where using my personal INTROOT
  • Brought the software to the shutdown state and:
    • restarted the CCC computer as this had crashed during the shutdown
    • The computer with the frame-grabber, optical, had also crashed. I left it in this state
  • System was brought to operational state using the OMC. Noticed that on DV01 not all components were running. Problem tracked to a bad entry in the CDB (Metrology.xml had incorrect permissions). Fixed this on both the gns and acc. Brought control to shutdown state, reloaded CDB, and back to operational state to fix this.
  • Everything fully operational by about 11:00am.
  • Successfully created an array using both antennas (using the CCL).
  • successfully created a SFI observing mode on both antennas (again using the CCL).
  • tried to get the current frequency.
    • Found problem with the WCA component not being operational. Fixed this by hand using object explorer, and later in the day, changed the code to ensure this never happens again.
  • Now getFrequency works and returns four values that are all between 100 and 110 GHz. The first two numbers were correctly around 103.8 GHz and the latter to are not meaningful as the associated LO2's are used for other purposes.
  • Noticed that the returned frequencies changed by around a few 10MHz. Tracked this down to the LO2's. Checked that fringe tracking was turned off. Pointed it out to Jeff Kern and assigned to Pablo to investigate. This may be a software bug.
  • Noticed that array tabs in the OMC were not being cleaned up (see COMP-2086).
  • tested various other functions in the LO observing mode that return other frequencies (like LO1 frequency, LO2 frequency laser synthesiser frequency etc.)
  • Re-enabled AMBManager access
  • At 1:00pm handed the system over to Jeff Kern for further ALMA-5.0.2 testing and to prepare for a demo to Fred Lo & Bob Dickman.
• CIPT: 13:00 - 18:00 ASDM and Delay Server
  • Set up system and began getting fringes, left in this state and investigating why it takes so long for changes in the cable delay to be visable on the CORR GUI output. Identified 2 reasons, resolved one and are working on the second.
  • ASDM work: We were able to start the Correlator from the Manual mode, but then suffered a failure in the Mount. Proceed with testing on Friday.

• PSI/ATF: Checking e-mails, google calendar, Journal. (Juan 2 h)

• AIV : 00:00 - 08:00 PSI scripts on 5.0.2 debugging. Due software problems downtime is 8h. [ebarrios]
• PSI/ATF: Cryo temps and pressures status in the morning: VA=164 PSI on Tank, AEC=85 PSI on Tank. (Juan 0.5h)
• PSI/ATF: We didn't have internet connection in some of the outputs in the main trailer. The problem was fixed power cycling the switch behind the red toolbox at the toolroom (Juan Gallardo 0.5h)

• Activity Log: - Active: CIPT(8h) Sci(3.5), Unscheduled: 0 hrs, Downtime: AVI(8h)+SCI(2.5h) + CIPT(2h) all due to SW.

January 22 Tuesday

• CIPT 22:00 The build finally finished, but we were unable to bring up the remote containers, apparently because of stale NFS mounts on the remote hosts. We consulted with Jeff by phone, but given that everybody was exhausted by this point, we gave up for the night and sent an e-mail to StefanoTurolla and PaolaSivera in Garching, asking them to check the configuration when they come in to work. (Joe Schwarz)
• CIPT 14:00 Joe & Gianluca start their work on 5.0.2 software. [ebarrios]
• CIPT 13:00-14:00 & 15:00-16:00 Jeff & Robert start the 5.0.2 mountPanel evaluation. [ebarrios]
• PSI tests by Peter 10:45 to 13:00. Results of tests includes: using this 5.0.2 software the spurious rail of 62.5 MHz harmonics previously visible in a CorrGUI^? TFB spectrum is no-longer present; the spurious 250 MHz harmonics reported by Science IPT in cross-power spectra are clearly seen and are removed by Walsh switching thus demonstrating that they originate in the electronics system ahead of the DTX Formatter.
• PSI 10:45 Finally Peter start PSI scripts tests (downtime 1.75h). [ebarrios]
• CIPT 10:15 Using /groups/user/Public_PSI_2/psi "re sus" work fine then Peter was able to start psi tests. But due to correlator problems subsystem was shutdown then power cycle OFF/ON on cdp and ccc then CORR containers re-started with no success. A new CORR subsytem shutdown then operational + Jeff action at the corrGUI fix the problem. The BB monitor MID was redefined for ALCATEL as 41 instead 2. [ebarrios]
• PSI 10:00 psi started but "re sus" fail (LORR x22 AEC NOT FOUND). Reported to Gene. Wainting Jeff for help. [ebarrios]
• ATF 08:45 Re-starting Acs. Took some time due to problems with correlator and control container. After 3 power off/on sequences Acs got the Operational state. [ebarrios]
• CIPT 08:15 Software version switch to 5.0.2 done by Jeff [ebarrios]
• ATF 07:55 Antenas unlock [ebarrios]
• PSI/ATF: Completed analysis of EDFA command failures http://jira.alma.cl/browse/BEND-94 - 3hrs Gene
• PSI/ATF: Confirmed AEC WMA / LORR anti-correlation in FO power level - 3hrs Gene
• PSI/ATF: Cryo temps and pressures look good. - Jack
• PSI/Corr: Measured EDFA rack and took images of Corr PC rack for Gene. - (.5-hours) Jack
• PSI/ATF: Went of inventory that Robert Ridgeway has in his office. - (.5-hours) Jack

• Activity Log: - Active: AIV(6h)+Corr(1h)+PSI(2h)+CIPT(8h) = 19h, Passive: 2 hrs; Unscheduled: 0 hrs, Downtime: PSI 2 hrs + 3 hrs from CIPT (all SW)

January 21 Monday NRAO Holiday

• 23:10MST, both antennas to Survival Stow; it was observed that the mount panel first go to STOP then recovers the TRACK and finally moves to park position. Of course, this tooks extra time on AEC but finally it moved. I terminated the mount panel process, the memory use readouts are (dv01 & da41): 233 & 258 Mb, and it looks like some memory had been released after the mount panel had been closed. The OMC presented a rare behavior or I just screwed it up at some point(?): for a while all containers went to red but system was operational; after a while I pressed ACS UP and now the system looks healthy as before. PSI scripts had been running the last ~ 8 hours, all looks good, with the exception of ARCHIVE that is still in PRESHUTDOWN. The OMC alarm panel had released some messages likely due to the end of the interferometry session on CRC-02. All looks fine now.[ Roberto]
• 21:48MST, moved back to 0730-116, fringes. Again it had been observed that VA 'stucks' in a position and then also moves slowly to reach the final requested position. AEC arrives presets faster. 22:20MST, Back to 0854+201, fringes. Back to 0927+390, fringes. 22:35MST, moved to 1058+015. 22:48MST, Moved to the high declination, faint object 1058+812. [Roberto].
• 22:28MST, more PSI alarms messages on DRX:

DRX    x141 COR 1                              704      METAFRAME_DELAY B
DRX    x141 COR 1                              704      METAFRAME_DELAY C
DRX    x141 COR 1                              704      METAFRAME_DELAY D
DRX    x141 COR 1                  N y y y y N y y      DFR_STATUS B
DRX    x141 COR 1                  N y y y y N y y      DFR_STATUS D

• 20:37MST, moved to 0854+201. Moved to 0927+390. ARCHIVE system back to ERROR and container down. It had been observed that the VA mount panel usually needs a second call (APPLY) before it gets the new coordinates and moves to its new position. When we moved to the present object, AEC mount panel shiowed the same problem. Memory usage: ~209 & 231 Mb, far below the 1Gb limit 'predicted' by Joe Schwarz. PSI alarms complains on " DRX x141 COR 1 N y y y y N y y DFR_STATUS B". 21:07MST, Back to 3C84, good fringes. 21:20MST, Moved back to 0530+135. 21:33MST, Moved to 0538-440, looks noisy but it is a ~ 12 degrees Elevation. [Roberto]
• 20:15MST, Joe and Gianluca are leaving. They asked to check the memory usage of ABM machines dv01 & da41. By now the memory usage is: 197.5Mb and 218.7Mb respectively. 20:21MST. Moved to 0730-116, it looks like faint fringes, apparently. Is clear but cold: T=-0.2C, P=788mBar, wind=3m/s, Tdew=-13.5C ~ 20:33MST, ARCHIVE ACC javaContainer down (red) and ARCHIVE subsystem in ERROR. Container restarted but sub system does not go into OPERATIONAL. In a second try (container to shutdown then try again) ARCHIVE just comes up to PRESHUTDOWN (from ERROR.) A _df_ shows disks are not full with the exception on "none" (/dev/shm). Log file 2008-01-22_03.34.29):

$> less .acs/commandcenter/ARCHIVE/ACC/acsStartContainer_javaContainer_2008-01-22_03.34.29 - reads....
41 52 43 48 49 56 45 5F 42 55 4C 4B 53 54 4F 52 45          ARCHIVE_BULKSTORE object is activated
2008-01-22T03:34:41.542 INFO [ARCHIVE/ACC/javaContainer] component ARCHIVE_BULKSTORE activated and initialized in 37 ms.
2008-01-22T03:34:41.764 INFO [ARCHIVE/ACC/javaContainer] Info message from the manager: Startup statistics: 4 of 4 components activated.
ping received, container alive. Memory usage 44589 of 52224 kB (= 2.2% of JVM growth limit 2019520 kB)
......
ping received, container alive. Memory usage 73065 of 188160 kB (= 3.6% of JVM growth limit 2019520 kB)

Might be related? At 03:44:24 UTC, OMC alarms displayed: "Scheduling:SchedArchiveConnAlarm:1" The message colour was and is stil RED and the Priority is also a red "1".

• 18:28MST, Back on BL LAC, can't see fringes or, noisy fringes. PSI alarms checked reports nothing special, Central LO Rack GUI looks happy and system is up. 18:42MST, moved back to 3C446. OMC-ALARMS reporting "Absolute phase of FTS lost at 01:39UTC, which might be related to the "absence" of fringes? Or it is not important? (How could I interpret the alarm message, I need a 'thesaurus'; PSI does not report nothing 'related' or critical.) Other OMC-alarms messages: FTS tuned to incorrect, Delay Command Executed. 18:59MST, Back on 3C454.3, fringes. Back to 3C84, fringes. 19:32MST, Back to 0359+509. Back on 0530+135. [Roberto]
• 16:36MST, pointing on quasar 1849+670, apparently not fringes?. Note that mount panel on VA shows (Summary) "On Target, ACU and Aux ACU pointing model not Applied, shutter OPEN (right) Metrology Not Applied and ABM pointing model Applied." For AEC, it says "NOT on Target, Shutter N/A, Pointing model (ACU, AUx, ABM) and Metrology Applied." 16:48MST, pointing on BL LAC, fringes. On the previous object WCA 1 (VA) was unlock but (I sword) the GUI looked different so it was not immediately corrected, which should explain fringes absence. My error, the PSI LLC script was not running! Now the 5 PSI scripts are running. 17:05MST, pointing on 3C446, fringes OK. 17:!6MST, back on 3C454.3, fringes o.k. Pointing on Quasar 3C84, fringes o.k. 17:46MST, Moved on quasar 0359+509, fringes o.k.Moved on quasar 0530+135, fringes.18:12MST, Back on 1849+670, fringes (faint but there it is.) 8:23MST, weather: T=3.5C, Wind=6m/s, P=786.2mBar, Tdew=-15.1C: clear.[Roberto]
• 15:10MST, Laing & Lucas, focus tests on both antennas while pointing on 3C454.3 and getting fringes. The test is: move the focus (using "PSI> rs move ANT ssr (0,0,z-value)" from -2500 to 2500 in steps of 500um for ~2 minutes and save the data using the CorrGUI file for later analysis. Tests ended ~ 16:23 local time, and for AEC we are now using focus-z=0 and for VA focus-z=-500 (PSI says is -484, there is an offest.) [Roberto]
• 12:10MST, Started the slow switching of sources with both antennas, using Nicolas B.'s script; during the two hours that we did this, we saw no container crashes or hangs. Wrote and archived a script to facilitate generation of thread dumps by the operator when containers go bad (crash or hang). We did notice that the amount of memory being consumed by each ABM container was increasing by about 20 MBytes/hour; this will need to be investigated, as we do not know whether this apparent memory leak is due to the container itself or one or more of its components. Since the container started at about 100 MBytes, it will probably take 50-100 hours of continuous operation before the container runs out of memory and aborts. Nicolas B. wrote a script to monitor container memory; it has been archived under ITS/ATF/debugScripts/src/GetMemUsage.sh. Turned the system over to Robert**2 for real-life exercising.
• 12:00MST, Antennas unlocked, tacking the software to an operational state.
• 11:30MST, Shutting software down and swapping to ALMA-5_0_1_13 -- this uses the ACS-6_0_4-B branch, which includes fixes to the logging system that have been back-ported from ACS 7.0. We are hoping for reduction or elimination of the incidence of ABM container hangs.
• PSI/AEC/VA: Cryos looking good, software is up and functioning! Yea! - Jack
• PSI/AEC: Finishing AEC HVAC recovery document. - (1-hour) Jack
• PSI/ATF: Gathered up all of the loose fiber patch cables on site and will clean and test them as time allows. - (1-hour) Jack
• PSI/ATF: Began sorting out the AEC antenna spare parts for storage. - (2-hours) Jack
• PSI/ATF: Installed temp data logger in the manhole to see if the sun has any effect. - (.5-hours) Jack
• PSI/ATF: Completed analysis of EDFA reset issue http://jira.alma.cl/browse/BEND-92 and worked on EDFA M&C issues - 4hrs Gene
• PSI/ATF: Completed analysis of EDFA monitor timeouts http://jira.alma.cl/browse/BEND-93 - 1.5hrs Gene

• Activity Log: - Active: AIV/PSI(5)+CIPT(2)+SCI(8) = 15 hrs, Passive: 7 hrs, Unscheduled: hrs, Downtime: SCI 2 hrs due to SW.

January 20 Sunday

• 01:20MTS, Antennas locally locked.
• 01:10MTS, System came back to Operational. Both antennas can be moved from mountPanel. RA & DEC are refreshing on VA mountPanel. System ready to be used and everything is in nominal conditions. Hector
• 00:30MTS continued with the recover procedure, I sent the ABM002 to startup but then could not connect AEC antenna to mountPanel anyway. Tried to Shutdown-Operational CONTROL subsystem but it never reached the shutdown condition. The fastest way to recover was to shutdown all the system. Hector
• 00:15MTS, AEC was presetted but didn't move. No red flag on ABM002 container and maciContainer were still running, so seems to be a hung rather than a crash. To recover, I tried to shutdown the ABM002 container, then the red flag appeared but maciContainer still was present. Ran the procedure and file was saved as ABM002-00:15_01212007. Hector
• 22:15MTS, was necessary to send the 'Apply' command twice on AEC mountPanel to get the values Az, El, RA & DEC, but antenna moved at the first attempt. This is not the same behaviour saw yesterday on VA. No news on VA, still RA & DEC not refreshing. Hector
• 22:00MTS, RA & DEC is not refreshing on VA mountPanel. The antenna seems to be in the commanded position (we get fringes) but Command and Actual RA & DEC display not change from RA 03:59:29.75 DEC 50:57:50.16 coordinates, which are the last refreshed ones. Then stuck on that. Azimuth and Elevation are refreshing OK. Hector
• 21:30MTS, Problems with the stability on VA FLOOG frequency. It went to zero without apparent reasons. Hector
• 19:40MTS, System tracking and getting fringes. Hector
• 19:20MTS, System handed over for science from software. Again problems with correlator gui, but shutdown-operational sequence to CORR subsystem solved the problem. AEC MountPanel^? doesn't show real status on "Summary". Focus posiition for both antena have not changed from yesterday(VA=-1000, AEC=0). Hector
• 09:MST, CIPT debugging of container hangs. Used script from Nicolas B. to move antennas back and forth, inducing container hangs or crashes every 10-40 minutes. Back-ported fixes to logging system from ACS 7+ to ACS-6_0_4-B; system will be rebuilt with this branch tonight.
• PSI/AEC/VA: Software down, manual cryo check shows systems ok. - Jack
• PSI/AEC: Studied the effect of the sunrise on the AEC fiber vault enclosure at Gene's request. - (1-hour) Jack
• PSI/AEC: Installed a temp data logger in the fiber enclosure at the base of the AEC antenna to study the effect of the sun.(.5-hours)Jack
• PSI/ATF: Cleaned up Alcatel trailer test bench and Gene's office and began organizing test equipment and cables. (2-hours) Jack
• PSI/ATF: Gathered up equipment requested by Nick Emerson for shipment to Chile. - (1-hour) Jack

• Activity Log: - Active: CIPT(10h)+SCI(6h), Passive: hrs, Unscheduled: (4.5h), Downtime: SCI(1h) due to SW.

January 19 Saturday

• 00:00MTS, Shut down software for Nicolas to do a build.
• 22:30 - 00:00MTS Continued to observe cycling between sources to check phase stability. No ABM crashes. System running smoothly.
• 22:30MTS, System handed over to Robert. It is tracking and getting fringes. Hector
• 22:10MTS, Again on VA was necessary to send the command "Apply" twice to preset it. Besides that, everything has been well. Hector
• 19:10MTS, Correlator gui didn't display any data. Shutdown CORR subsystem and send back to operational solved the problem. System is tracking and getting fringes. Hector
• 18:15MTS, Antennas preset. Two times happened that to send preset to VA was necessary to press twice the "Apply" button on mountPanel. At the end both antennas moved. Focus position for VA is -1000 and 0 for AEC. Hector
• 18:10MST, Swapping back to 5.0.1 for Hand off to science. (will leave an intlist with debugging commands)
• 15:30MST, Debugging of the abm freezes and crashes. Some diagnostic code provided by Gianluca, Nicolas B. and Joe was installed along with a patched MountController provided by Ralph. We saw segfaults on both the Vertex and the Alcatel antennas, apparently in the MountController's trajectory handling. A gdb snippet was sent to JorgeIbsen. We did not see any instances of zombie containers (container alive but unresponsive).
• 15:21MST, ITS tools failed to build what they were commanded to. Defaulting to HEAD-2008-01-15-RTLOG which contains the rtlog in INFO level.
• 14:53MST, re-configure deployment so we are in the default configuration.
• 14:43MST, The CorrGUI failures referred to below (i.e. ones associated with CORBA timeouts during CDP_MASTER component load) are all accompanied by kernel Oops of the rt log level variety.
• 14:41MST, CORRELATOR saved data into ARCHIVE (asdm files).
• 13:30MST, Still dealing with error trying to produce ASDMs.
• 11:00MST, Correlator subsystem is started without timeout, corrGUI works.
• 10:40MST, re-arranged software deployment so correlator diskless computers mount ACSROOT from crc-01 (operator console).
• 10:15MST, Software was started with the new deployment (all diskless mounting ACSROOT from gas). We saw the CORR timeout and CORR gui failed to work. J thinks this it is due to the CORR time out.
• 09:15MST, re-arranged software deployment to be able to run software while building rtlog patches.
• 09:15MST, rebuilding parts of HEAD-2008-01-15 without the rtLog patch.

• PSI/AOC: Went to Gene's office at his request and retrieved instrument cases, manuals, and LO parts. (3-hours) Jack
• PSI/AEC: More of the antenna lockout switch came apart and we can turn the antenna switch on and off but can't lock it out. I'm ordering replacement parts. Tried to repair it without success. (1-hour) Jack

• Activity log - Active: 11 hrs (development) 6 hrs (sci), Passive: 7 hrs, Unscheduled: 0 hrs, Downtime: 0 hrs

January 18 Friday

• Sci:
  • atf science obs summary for 18jan08
  • Todd's drawing of the ATF LO layout (as of Jan 2008)

• 00:45MST, AEC lock too broken so I used both ee-stops, I mean, that located close to the lock and that located at the opposite side of the pedestal, just in case. [Roberto]
• 00:15MST, fringes on 1058+015, problems on 0854+201, the VA WCA was unlock but does not explained the failure. Pointing models verified, all was o.k., not clear what happened on 0854+201. AT the very end (00:30MST), ABM001 crashed again, mount Panel was in unknown state. Apparently this happened once I sent ALMA02 to Survival Stow, disconnected the Object Explorer, then noted ALMA01 was not in TRACK but unknown; recovered and sent it to park position. Also, QL passed to INIT state. Also, it was observed the runOMC session under VNC still does not display 'alarms' but a local session does it!And, the failure also created again the message "Error executing command" that is deleted only with "xkill". Antennas in Park position and PSI scripts already running. System up but QL untouched for CIPT. [Roberto]
• 22:10MST, ABM002 fails soon after the restart (mount display UNKNOWN state) but this time, we disconnected the mount panel, used "ps -u almaproc" and verified the maciContainer process was already off, then we just restarted the container, and using the OBJECT EXPLORER we did a refresh and then listed "By Device" and searched under CONTROL -> MASTER and then found the "reinitializeAntenna(String)" method and invoked it using ALMA02 and we recovered the system. 23:35MST, failed again on VA, they moved 1 degree off in declination (from -5 to -6 degrees) around Orion, AEC moved but VA did not. After some unfruitful tries the workaround was kill the container, wait, check the maci container process was gone, restart the container, use the object explorer, make a search (refresh), look under Devices for CONTROL/MASTER and then 'reinitialize antenna' using ALMA01. And this had worked! Great! Looks like VA had been particularly unstable this night. As extra observation, after every short or long reatart (failure) AEC always starts in STOP mode but is not unusual that VA 'restart' in TRACK mode, almost magically! Once again it had been observed the differences in the Status message in AEC versus VA: if the system is running in both mounts the pointing model is loaded, but when _VA fails_ besides the NOT ON TARGET message, it also says the pointing model is not loaded. And, under details, instead "status O.K.", it says "Invalid Mode Change". [Roberto]
• 21:15MST, we are restarting the system: the sequence was move to quasar 0359+509, AEC moved but VA was 'frozen'. After some tries with the mount panel, we tried with PSI "rs move" and then we saw the container changed to STOP (red.) Summarizing, the ARCHIVE was ONLINE, it finally went to SHUTDOWN, the ARCHIVE/ACC/cppContainer went into RED, "then" the CONTROL/ACC/javaContainer also failed (and it was not possible to recover it!), Control/AMBSocketServer went to RED so here we used the procedure: the component CONTROL/AmbSocketServer was "Force(d) system-wide deactivation, then we sent a SHUTDOWN request on CONTROL/AMBSocketServer, wait a while, restarted that container but, because we did not putted the CONTROL sub system first into SHUTDOWN, the CONTROL/AmbSocketServer sub system was in UNKNOWN status and then the trick was search again the CONTROL/AmbSocketServer and use the " Have Activated" command which after a while sent the subsystem AmbSocketServer^? back to OPERATIONAL. But, given we were not able to recover the CONTROL/ACC/javaContainer and the mounts remained in Delay Getting State, it was finally needed to restart the system. After the restarting we sued ""PSI>c sl mpr=85.8" so we had set back the Slave Laser photonic reference at 85.8MHz. By now the ARCHIVE is OPERATIONAL and we are on quasar 0530+135 but no fringes yet. [Roberto]
• 20:35MST, Nicolas Barriga reviewed and recovered the ARCHIVE subsystem, I understood he did not found a clue and by now this is OPERATIONAL. We observed by ~ 20 minutes on RA= 04h23'15.8" DEC=-01d20'33.065". Now we are back on Orion (the previous spectra looks great according to Tood's reduction.) 20:40MST, ARCHIVE backs to ONLINE. We will leave it there since is not critical for the interferometry.[Roberto]
• 18:00MST, tests of VA focii, now is working. The optimal focii values are VA=-1000 and AEC=0. The temperature was around -4C and the object used was 3C454.3. Then, fringes on 3C454.3, then by 19:00MST used "PSI>c sl mpr=80.0" for setting the LO frequency and then "c wca lock=(80.0,6)" and then moved to Orion. A different experiment, while on Orion, they applied a 1 degree extra in declination on both antennas. Around 19:34 it was found the ARCHIVE system was in ERROR and the ARCHIVE/ACC/javaContainer was in STOP (red) mode. The container was restarted and the system putted first in SHUTDOWN then in OPERATIONAL. HOWEVER, 10 minutes later is complaining the subsystem is now in ONLINE. We putted it back to Operational.... 15 minutes or so it was INIT, tried to put it back to OPERATIONAL, it says, "reinitializing". [Roberto]
• 9:30-13:00. PSI and BE (Peter, Robert, Drew and Sylas) worked on the new 4-12 GHz IF system. Main conclusions: the PSI script for switching the IFDC to the USB input does not work as expected; the Labview and CCL IFDC commands work correctly; there was a cabling error in AEC which was corrected; both antennas have good sensitivity on the 4-12 IF with an LO1 of 80.0 GHz provided that an IF at the top end (approx 10 GHz) is used. This work shows that it should be possible make the 4-12 GHz IF work for 86 GHz observations.
• 9:00 CIPT is testing the integrated system attempting to get data through the system to the Archive. The Correlator continues to function properly no sign of the failures observed early in the week.

• PSI/VA: The subreflector is showing a fault with hexapod collision switch #1. This will require troubleshooting the sensor and wiring at the apex. The problem is, the weather is very cold and more problems could be created with brittle wiring and connectors. I'll point the antenna near the sun to try and warm up the hexapod area before troubleshooting. - Jack - After a short time warming up the subreflector, the fault went away. Checked cables and connectors at the apex and tried to exercise the subreflector but it doesn't respond very fast probably due to the cold. I will exercise it again later today. - (1.5-hours) Jack Later.....
• PSI/VA/AEC: Both cryo systems are looking good and maintaining oil temperatures during cold overnight conditions after the fan mods. Jack
• PSI/ATF: Shipped 3 more boxes of bin module parts to Bill Shillue in CV. (1-hour) Jack
• PSI/ATF: Inventoried CLO test equipment for Dick S. (.5-hours) Jack
• PSI/AEC: Charged He compressor (GM side) with cryo oil. (.5-hours) Jack
• PSI/VA: Lost CAN communications with the PTC and had to reboot it and the ACU. - Jack

• Activity Log: - Active: 13 hrs (development), 7 hrs (sci), Passive: hrs, Unscheduled: 4 hrs (after build), Downtime: 2 hrs (1 SW + 1 HW)

January 17 Thursday

• Sci:
  • atf.science.obs.18jan08.doc: atf science obs summary for 17jan08

• 22:MST, Robert Laing had done some tests offsetting the antennas 30" in EL and then in AZ for 2 objects at low and then at high elevation. The, VA fails again, mount panel frozen while trying to move to quasar 0359+509. It does not show any message this time. After a while, the container is down (red) but the maci Container is running, there is no answer at the "getEpoch()" method on Object Explorer. Is confussing but apparently the system is on HARD tics. PSI complains about VA but the low level call "teHandler..." show there are HARD tics both in VA as in AEC. Or in other words the status is the same in VA as in AEC, but the container ABM001 is down. Also, the same that in the first VA failure, we got this annoying little window _Error Executing Command_ with not any extra comment or useful information. And the only way to get rid off of this message is using "xkill". We are now making a full shutdown of the system in preparation for the new build CIPT have to start by now. [Roberto]
• 20:20MST, We are now on a new source (RA=10:43:09.04 DEC=24:08:35.41.) A new problem/observation: in the near past, the PSI command "c ifdc zb" worked on both antennas. Now it works only on VA. Using "c ifdc antenna sg=(x,y,z,w)" this displays the _IFP channels 0X29A and 0X2A for AEC_ but it acts on 0x2A, which looks -to me- new and rare. Then, we can't tune the gain on AEC as we did on the very near past. This works on VA, then we can see 0x29A but I underline on AEC we can see 0x29A and 0x2A. Robert Laing suggested to move the AEC up to 50deg EL and then back to source, this improved a bit the signal but is still noisier than VA (~ 50mV in AEC, ~ 5mV in VA). Also, the VA baseband signal shows the negative slope signature of a source at the East but AEC looks more like a 'ratty' signal. Also, the LO optical power (RX_OPT_PWR) on AEC is ~ 0.07mW, down below the range already defined as 'good'. Jack says this module could go so low as 0.02mW and that a real failure would be reflected in the absence of Hard tics for AEC/ABM002. So we did not touched the module for now.[Roberto]
• 19:30MST, after (or through in fact) the test on the pole, VA started to fail. When we moved back to the quasar VA did not moved, it looked frozen. PSi was able to move the VA but after a while, the ABM001/cppContainer was down. We tried to recover just that machine, it worked but then when we finally moved VA AEC (ABM002/cppContainer) was down. In the recover procedure for AEC the ABMBSocketServer^? did not comeback so it was needed to stop the GUI's and recover that component and container. But then, once we have all up again, we opened a mount panel and then ABM002 failed again (ABM001 had already failed while we were recovering the AMBSocketServer^?!) So, general restart of system. And then it had been observed -again- that pressing the SHUTDOWN button just affects Scheduling and Control but up to the "pass_1". So you have to force the shutdown or go directly to the "ACS Down". We are not checking all machines every time but "apparently" 'maci containers' disappear when the container fails but also apparently, always the kernel modules are unloaded and it is needed to unloaded them or, shortcut, pstrip the machines. Downtime, not less than 40 minutes.[Roberto]
• 18:30MST, some tests were done moving the focus on AEC and VA. The VA sub reflector apparently is not moving, PSI reports it does not move to the requested position (z=0.) It stays in its default position (z=-484.) AEC sub reflector does moves and it is possible to see an effect in the phase of the cross correlation displayed in the corrGUI; what we saw suggest a better focii value for AEC is ~0. Robert Lucas points the foci have to be a function of temperature and elevation. Then a different test: we had point both antennas to Declination 90 degrees and they are saving a new CorrGUI file with data gathered on this "black" spot (in the sense of no known source of signal.) The idea is to collect data about the features of the hardware that are affecting the real science data. Also, it had been observed the status displayed by the mount panel for AEC are "the right ones", in AZ/EL, RA/DEC (well, in fact RA is changing which have but it says there is no pointing model loaded (which is wrong) and, by the other hand, the mount panel for VA is wrong in AZ (it claims -92d25') and in Dec (it display 16d08' instead 90degrees.) VA recognize the pointing model is loaded.Also, it was observed for a while the status of VA changed from "track" to "unknown" and then it got back to "track". Looks unstable.[Roberto]
• 17:40 MTS. System is ready to start science. AEC subreflector moved without problems, but seems that VA subreflector cannot be moved. Now tracking on 3c454
• 10:30 MST CIPT work on Correlator undet ALMA-5.0.2 J. Perez has arrived and the correlator is now behaving correctly we are able to get data from the Correlator to the Corr GUI. One of the container "freezes" happened and Joe was able to investigate the details of the crash and send them off to Garching for further analysis.
• 03:30 MST. Shutting down after a 9 hour track on 0359+509. Phase plot Amplitude plot. Data file = Spectra_01-17-08_1.21.38.779.dat. No downtime - Todd
• Sci (Debra for Todd based on an e-mail he sent out): Another smooth night, with a 9-hour track completed on 0359+509. I managed to add temporal vector averaging to the analysis script, and am also plotting amplitudes. Enclosed are 3c454 scans from Monday and Tuesday nights. The LLC correction script is not yet foolproof, as it doesn't remove all of the jumps in the second track included. However, the first track shows an rms phase of ~3 deg in the flattest hour (when binned to 1 minute integrations). You can also see the amplitude drop sharply when the source goes below ~10 deg (don't have the elevation easily available to annotate it).
• PSI/AEC: He compressor fan controller mod performed very well overnight and I will modify the VA He compressor fan controller today. This will eliminate the need for any low tech cardboard. - Jack
• PSI/VA: Reprogrammed He compressor fan controller and relocated temp sensor. Added a small amount of cryo oil. (1.5-hours) Jack
• PSI/ATF: Continued packing bins and module parts for Bill Shillue. Will ship 3 boxes of bins and module parts on the noon shuttle today and probably two more boxes tomorrow. (1-hour) Jack
• PSI/AEC: Troubleshooting shutter status, PSI script says its not open or closed. A visual inspection from the manlift shows that the shutter pinion gear has come off the rack gear. Later today when an operator shows up on site, I'll have them assist me in getting the shutter back on track. Later......Hector and I put the shutter back on its track. (.75-hours) Jack, Hector
• PSI/VA: Investigated why the HVAC may have shutdown on Monday. Found FA damper had vibrated closed and this may have caused the compressor to shutdown on high pressure. Fixed the damper in position with hardware. (1-hour) Jack
• PSI/AEC: Working on HVAC recovery manual.
• PSI/ATF: Debra had a great idea to adjust the one remaining webcam on the whole ATF site. Duh! Will do first thing tomorrow morning because the Sun is directly into the camera this evening.

• PSI/ATF: Continuing with test equipment inventory for Dick S.
• PSI/AEC: Ordering antenna lockout switch parts.
• PSI/AEC: Readjusted the fan controller to a setpoint of 5C so the fan runs longer when the ambient goes higher. Just want to test this setting. (.25-hours) Jack
• PSI/ATF: Packed up three remaining boxes of bin modules for shipment to Bill Shillue in CV. - (1-hour) Jack

• Activity Log: - Active: SCI(8h)+CIPT(9.5)= 17.5 hrs, Passive: hrs, Unscheduled: PSI(5h), Downtime: SCI(1h)

January 16 Wednesday

• Sci:
  • atf.science.obs.17jan08.doc: atf science obs summary for 16jan08

• Started using the system at 9:30am (8am start was delayed because of problems with the switch to ALMA-5.0.2).
  • Found the CDB checker was complaining that CDB entry for the Metrology component (PTC) was wrong. Still the wrong value appeared to work so I used it.
• Switched the alarm system to the ACS implementation. Tried swithing the control subsystem between shutdown and operational states more than ten times. Things did not crash at shutdown (unlike the tests I did 6 days ago).
• Found the ambmanager was somehow triggering a crash at shutdown. I disabled it for the rest of the day.
• Switched back to the CERN Alarm system implementation and did the same test. Things still worked and I do not understand why.
• Repeatedly created and destroyed an array and a Single Field Interferomery observing mode. Checked all components started and stopped as expected. I used a simulated correlator subsystem for this test.
• Tried to understand why only one LO2 component was being started. Finally understood it was because I needed to restart the acs container daemons to pick up the new configuration (INTROOT). Did this and then repeated the abovementioned tests (but only a few times as I was runing out of time).
• Handed the system over to Jeff Kern for a quick test at 4:10pm. He then switched back to ALMA-5.0.1 for the SCI IPT users.

• 23:30 MST. ambient temperature at VLA is +11F = -12C
• Sci (Debra for Todd and Peter based on an e-mail from Peter): Todd and I (Peter N.) were just looking at the amplitude of the cross-correlation spectra - see attached. All of the Channels 64, 96, 128, 160, 224 that have anomalies that we were thinking were FFT problems appear to have spurious signals in them. So the problem is probably not an FFT issue but rather a problem of spurious signals generated by the system. I am suspicious of the DTS system because the spurious signals are at

harmonics of 125 MHz, but we will have to investigate further. * CrossCorAmp.pdf: Figure: amplitude of the cross-correlation

• Sci (Debra for Todd based on an e-mail he sent out about the data collected on Monday): The structure in Antonio's plots constructed from an 8-channel average were due to the presence of channels 127-129 in the average. There is a phase ramp in these channels. This is likely due to the fact that channel 128 is half-way through the list of channels in FFT, i.e. a breakpoint where odd things tend to happen if you don't have the details correct. Following this discovery, I decided to plot the timeseries for all 256 channels during a 5-hour transit-to-horizon track on 3c454. The 256-page PDF is 29 MB in size, and can be found here: oper01.atf.nrao.edu:/groups/sci/interferometry/20080114/timeseries256.pdf. Paging through this, you can see which channels are affected by beacons, birdies or other anomalies. Here is the list of suspect channels:
  • 0,1, identically zero
  • 30-36 beacon?
  • 64-66 2^6 FFT problem?
  • 96-97 96 is another interesting number = 64+32
  • 127,128,129 2^7 FFT problem?)
  • 159-161 note 160 = 128+32
  • 179-181 no signal (just noise, very weird)
  • 223,-225 note 224 = 256-32
  • 255 0/180/0/180 repeating

I will now avoid these channels in the spectral vector average script. In the meantime, I will try to implement a temporal vector average that will follow the application of the LLC correction that Antonio has developed. I think this file should be forwarded to the correlator folks (I have CC'd Jim Pisano). I would like to thank Antonio for putting us on the track of examining these details.

• 22:00 MST. System handed over to Todd. It is Still tracking on 0359+509 source. Hector
• 20:57 MST. Jack reports that the He compressor low temperatures are looking good after his fan modifications. - Todd
• 19:00 MST. The PSI alarm page reports ABM001 is not on hard ticks. However, Hector has demonstrated that this is due to the name change to dv01. The alarm script is ssh'ing to almaproc@abm001, but this name is no longer recognized. If you manually ssh to dv01 and give the command /groups/users/TE/teHandlerGetTime, you can see that it is still on Hard ticks.
• 19:00MST, A short experiment shows the LST displayed by the OMC now compares quite well with an independent value provided by the web site "http://www.jgiesen.de/astro/astroJS/siderealClock/", so, as Preben Grosbol said, this problem now looks fixed. [Roberto]
• 18:22 MST: on source 0359+509 at LST=01:55 with fringes
• 18:00MST, It had been observed that a VNC session already running on _oper01_ is killed if a second VNC session opens from a different computer, both on Windows as on Linux. This poses a limit -apparently- to the use of VNC as a remote monitoring tool. May be I am using the wrong address or port?[Roberto]
• 17:40 MTS Software ACS-6.0.4, ALMA 5.0.1 version. System is started up for science. PSI was configurated for 5.0.2 version. Jeff was called and changed to 5.0.1 configuration following the Gene procedure sent by mail. Now we are tracking on source. AEC Shutter still in intermediate position according to PSI. Also according to PSI still NOT ON HARD TICS present on VA, but doing a ssh dv01-abm>groups/user/TE/teHandlerGetTime the reply is HARD. Hector
• 12:00MST, mice traps review in AEC, bait untouched, no mice. [Roberto]
• PSI/AEC/VA: 13:30UT, software not available yet, checked cryo systems manually and all is well. (.5-hours) Jack
• PSI/AEC/VA: Will reprogram He compressor fan controllers today and see if this will keep the oil warmer when the ambient temps dip to very low levels. Tonight we are expecting temps to dip to 2-degrees F.
• PSI/AEC: Reprogrammed the He compressor fan control and relocated the temp sensor to the GM oil heat exchanger output. The fan now cycles on and off with 15 degree C oil temp. Tonight I'll monitor the temps as the ambient drops and adjust the hysteresis to maintain temperature of the oil and prevent shutdowns. If this works, I'll modify the VA He compressor. (2-hours) Jack
• PSI/CLO: Packed up bins and module parts to ship to Bill Shillue in CV. (2-hours) Jack

• Activity Log: - Active: 14 hrs (development) 6 hrs (sci), Passive: hrs, Unscheduled: 4 hrs (after build), Downtime: 0 hrs

January 15 Tuesday

• 23:00 A new plot of Monday night's data on 3c454 which shows how the application of the fringe count in the LLC log removes the phase jumps. (A text file with the corrections as a function of integration number was compiled by Antonio. Such files are now read by the avgchannelsllc.py script.) - Todd
• 22:30 A new entry in the Science IPT's Interferometry Checklist has been made describing how to perform Vector averaging and LLC corrections to interferometric phase data using python scripts from within casapy. - Todd
• 5:30-10pm Sci (Debra for Todd based on an e-mail he sent out): Tonight we obtained a 4.4 hour track on 3c454. Phase data are attached. The second attachment shows Monday night's data on 3c454, and how the application of the fringe count from the LLC logs removes the phase jumps. (A text file with the corrections as a function of integration number was compiled by Antonio. This kind of file is now read by the avgchannelsllc python script, the usage of which is now documented in a new Sci IPT inteferometry checklist.) Regarding down time, there was none. However, about an hour before the end of our timeslot, Jack noticed on the PSI alarm page that ABM001 was NOT ON HARD TICS. Fringes were still present. We called Nicolas and he noted that the usual method of fixing this situation is completely restarting the system. Since so little time was left, we decided not to. Computing took over at 10PM to begin a software rebuild.
• 22:00 Computing took over to begin a software rebuild. - Todd
• 21:50 Closing down interferometry and locking antennas. The lock has been left off of the broken switch on AEC as per Jack's instructions. 4.5 hours of data recorded on 3c454. Data file is Spectra_01-16-08_0.24.58.232.dat. Here is the phase vs time plot with vector averaging alomg the spectral axis. - Todd
• 21:00 Jack noticed that ABM001 was NOT ON HARD TICs. Fringes were still present. We called Nicolas and he noted that the usual method of fixing this is restarting the system. Since so little time was left, we decided not to try it. - Todd
• 19:00 System handed over to Todd. It is Still tracking on 3C454 source. Hector
• 17:00 MTS Back to previous ACS-6.0.4 version. The status of AEC Shutter on PSI says that it is in between, but visually it look open. So continued with the setting up for the system to get fringes. Now is tracking on 3C454 with no problems. Hector
• 13:00 MTS Jeff and Robert is working on CORR software with the ACS-7.0 version
• 09:00 Peter and Robert worked on the 4-12 GHz system until 13:00. Found that the PSI script used to switch the input of the IFDC's to the USB input does not work correctly on either antenna. This explains why we have been unable to switch the 4-12 GHz IF into the basebands. We will try CCL or Labview commands next time. Also found that with the standard LO1 value of 85.8 GHz the 4-12 GHz IF has poor sensitivity, presumably because the FE has a double-sideband response for this value of LO1. The sensitivity of the 4-12 GHz IF improved when LO1 was lowered to 80.0 GHz which is the value required for 86 GHz sky frequency.
• Cooling system were found OFF on both antennas. Temperature on 4K FED sensor on VA was 158 K and on AEC 169 K. Hector
• 09:00 MTS ARCHIVE/ACC/JavaContainer was found down and ARCHIVE subsystem in ERROR status. Container was sent up but subsystem never went to Operational. Shut down and Restart everything brought back the system without CORR/CDP containers Up. Had to Shutdown CORR subsystem, shutdown all CORR containers, pstrip corrps cycle 1, pstrip corrps cyle2, restart CORR containers and send CORR subsystem to Operational. Hector
• PSI/AEC/VA: After investigating the PSI archives on the FEC's it was discovered that when the outdoor ambient air temperature dropped very low (<10 degrees F) the viscosity of the oil changed to the point that it would not flow well out of the heat exchanger and therefore the GM compressor would overheat and shutdown. Tonight I experimented with some cardboard patterns to block air flow across part of the GM heat exchanger to keep the system functional. Tomorrow I will reprogram the fan controller so that the fan cycles with temperature. (2-hours) Jack

• Activity Log: - Active: 7.5 hrs (Sci) 10.5 hrs (development), Passive: 6 hrs, Unscheduled: 0 hrs, Downtime: 0 hrs

January 14 Monday

• Sci (Debra for Todd and Antonio Hales based on an e-mail exchange that was sent around) - subsequent analysis of the phase in different channels suggests that some of the channels have bad phase (when you include them in the chanel average, the phase looks much worse). Todd will follow up.
• Sci (Debra for Todd Hunter, based on e-mail from Todd): The system ran smoothly tonight. We completed a 5-hour run on 3c454, followed by a 5-hour run on 0359+509. Data files are Spectra_01-14-08_23.53.37.354.dat, and Spectra_01-15-08_5.12.51.631.dat. In between maps, I checked the y-factor on hotload/sky and got ratios of 1.76 for AEC and 1.44 for Vertex. So the receiver in Vertex continues to be the worse of the two (although the rms in the total power datastream is worse on AEC). I wrote a python script that reads the CSV file output by the CorrGUI parser into a CASA table, and then performs a vector average over most of the channels (225 out of 256, avoiding the birdies). Finally, it generates a plot and writes a new CASA table with a factor of 256 fewer rows. Here are plots comparing a single channel of data with the average data: first plot = 3c454, second plot = 0359+509. Next, I hope we can incorporate the corrections for the LLC jumps (from Antonio's work) into the averaged data to see how well the segments will then join up.
  • Spectra_01-14-08_23.53.37.354.dat.pdf: Figure 1: 3c454.3 fringes (Todd Hunter)
  • Spectra_01-15-08_5.12.51.631.dat.pdf: Figure 2: 0359+509 fringes (Todd Hunter)
• 20:00 System Handed over to Todd. It is Still tracking on 3C454 source. Hector
• 15:00 Jeff got fringes and teach to Todd how to setup the correlator. Antennas Continuing tracking on 3C454 source. Hector
• ATF : 14:30 MTS Operators Cross training. Hector
• 13:30 MTS Power on FE Rack was OFF probably because the warm on the cabin (HVAC problem). Tryed unplugged the power suplly but didn't work. Solved switching OFF-ON the power strip. Hector
• 13:30 MST BE's Stian and Nick replaced the AMBSI1 board on the PSD in the VA since it had failed earlier. The earlier failure involved the temperature sensor being stuck at 22 degrees. This failure caused the digital rack to NOT shut down under a high temperature event in the VA receiver cabin from an HVAC failure. Tested to work by Drew
• 12:00 MST BE's Stian and Nick installed a PSD into the CORR PC rack to give power cycling ability to the EDFA/FOAD units atop that rack. This moves the DC power from the CORR to the PSD in the CORR PC rack. The PSD was tested to operate as designed by Drew.
• 10:45 MST BE's Drew reset the VA compressor to get the VA HVAC to start cooling the receiver cabin down again after it failed over the weekend.
• 12:00MST, it had been observed the VA mount still shows that probklem in which a command from, say, the mount panel, is "overwritten" by a position previously stored in the ACU. If we sent the VA to maintenance, the VA comeback after a while to AZ=90 and El=30 (this was the specific test-failure of today.) This had been already reported in _JIRA ticket COMP-1517_ . I readed the JT and it reads "resolved" but as long as I can see, the failure is still there. According to Ralph this was fixed on ALMA-4.1.4 so we ought to say that apparently it does not work or system still fails on ACS-6.0.4 Build:ALMA-5_0_1_12 [Roberto.]

• Activity Log: - Active: 5 hrs (training, hardware) 9 hrs (sci), Passive: 10 hrs, Unscheduled: 0 hrs, Downtime: 0 hrs

January 13 Sunday

• 23:00MST, system handed to Jorge and Pablo for their CIPT tests.
• 22:40MST, After the tests on VA and AEC Todd had requested a FivePoint^?. We tried Mars but we can't see it (?). I mean, doing a 1 degree offset, it is not possible to appreciate a change in the signal level. Then we started the Five Point but it halted at the beginning. We tried trice, every time it was needed to kill the process. It was clear then the mount was not moving. We tried to move to a point in AZ,EL but did not worked. We tried with PSI and the system did not answered, it reported a CIPT error. Then the mount claimed Delay Getting State as it did in previous failures. We tried restarting the full system, in between we had seen again a "new" message: _Error Executing Command_ It looks like a dialog window but or the content is empty or graphically hungs so it displays no content. In fact, it disappears just with _xkill_. After the restart, still we can't see mars, I mean, Five Point does not show the characteristic square pattern and an offset does not show a significant signal variation. SO, we are back on Saturn, an offset shows a variation but the FivePoint^? does not show anything. Is there any issue with the electronics? [Roberto]
• Sci (Debra for Todd based on an e-mail Todd sent out): Here are the results from tonight's tests. In summary, I think the mechanics of the pointing model are correct with respect to high elevations. However, I believe there is a software limit in place at el=88.96 degrees when observing in equatorial tracking mode. First, we manually commanded the AEC antenna to a series of discrete elevations between 89 and 91 (and azimuth=90). After the antenna acquired each position, we ran PositionStreamClient briefly to record the commanded, actual, and encoder az/el's. I then plotted these discrete data points on top of the theoretical pointing model curves from your applyModel() subroutine (see the first plot attached). The match is excellent, and as expected, there is a "plateau value" present inside the 2.5 arcmin keyhole (vertical red lines). To check whether the behavior of the strong azimuth curve is sensible, I computed the great circle distance from a fixed position at the horizon (az=90,el=0) and plotted it against the commanded elevation. (For the calculation, I used the formula on this page: http://en.wikipedia.org/wiki/Great_circle_distance, which is accurate for all separations.) The result is a straight line with unit slope up to the keyhole, which tells me that the pointing model is following a rising source as expected. Some oddness happens when elevations above 90deg are commanded. Instead of the separation continuing to increase, it decreases, because the antenna has moved by ~+180deg in azimuth but it is still going up over-the-top in elevation. This suggests that over-the-top observing will not work in the present implementation, but perhaps this is not in the ALMA spec. By the way, we were unable to command elevations >= 91:00:00 from the mount panel (an error box popped up), but 90:59:59 worked. Next, I thought it would be interesting to see how high in elevation that the antennas could follow the pointing model when tracking at sidereal rate. First, we tried to track an RA/Dec position which transits along a tangent to the keyhole radius of 2.5 arcmin. Both antennas remained "On Source" until elevation 88.96. The commanded elevation then stopped increasing, despite the fact that the pointing model wanted to send it higher. (See the second & third plots attached.) There seems to be a software limit somewhere that prevents tracking to elevations higher than 88.96 when tracking in RA/Dec mode. We confirmed this limit by tracking sources through transit on both antennas. Positions that transited at el=88.90 were successfully followed throughout transit (i.e. to the unwrap point), while those transiting above 89.0 lost acquisition before transit. Thus, at present the effective keyhole radius (in software) is ~1.04 degrees. Although the pointing model keyhole of 2.5 arcmin works, currently it will never be encountered in RA/Dec mode due to the more stringent limit located somewhere else. Finally, since radio pointing had not been attempted for a month, we briefly tried to observe Mars (180 Jy) and Saturn (220 Jy) with AEC. We did not see Mars at el=81 but did detect Saturn at el=15, though not strongly enough for a successful FivePoint^?. Since it is now 2008, I wonder if the ephemeris could be out of date for Mars? Anyway, I'm sure we will try this again tomorrow.
  • aec.radectrack.pdf: Figure 1: AEC RA-Dec track
  • aec.separation.pdf: Figure 2: AEC separation
  • vertex.radectrack.pdf: Figure 3: VA RA-Dec track
• 19:00MST, The VA presented the same failure as in AEC and the following sequence of events had someway been registered: close to 88.6 degrees elevation Todd saw the Position Stream Client reporting _Off Source_ , the mount panel was still tracking, Roberto saw the mount "shaking". Todd says this might be normal as the servos have a 10Hz refresh rate and then, after 0.1 seconds, the mount try to reach the new position which in turn is changing quite fast. Then finally the mount failed with the _Axes Disagree_ message. Todd believes a different experiment have to be tried using a Declination a full minute smaller that the latitude of this place, this is, use ~ 33:01.40 declination. The procedure of recovery after the failure takes too much time, say, considering that the Axes Disagree failure means the AZ had failed but the EL remains in its AUTONOMOUS state. In this new "lower" declination position, the mount wraps as usual (did not 'fails'). The same test is then repeated for an even lower declination, 31 degrees 46 minutes. Then the idea is to fill the gap of declinations up to the point in which the mount fails. [Roberto]
• 18:45MST, same failure of AEC mount, _axes disagree_ so we will try the test using the Vertex antenna instead AEC. The failure happens with >~88 deg EL and the mount ~3 minutes before culmination.
• 18:30MST, slight mistake, after the last failure we didn't loaded the pointing model! Damn. Instead having the Pointing Model status under summary (as it is now in the mount panel) it could be a colored button on the mount panel with a _PM_ in front. Red might be unloaded, green loaded.[Roberto]
• 17:58MST, we tried to follow a point starting ~6 minutes before culmination but the mount panel goes into _Axes Disagree_ , the AZ axis was in shutdown and EL remains AUTONOMOUS which is a kind of known issue for AEC (there is a JIRA ticket AECANT-19 since this failure had been already observed November 2 2007 and a number of other times.) The mount also display a message stating it can't reach the requested position. For the recovering this time it worked to put both CONTROL and SCHEDULING in SHUTDOWN, stop the ABM002/cppContainer in shutdown, clean the machine of the unloaded kernel modules (it was also verified the _maci_ container was already stopped and there were not _ops_ .) Then it was possible reconnect to the mount panel but, for the sake of the security, the VA mount will be also used since the electronic signal (baseband) is not needed.[Roberto]
• 16:30MST, Todd Hunter starts his tests using AEC (VA unavailable because of Gene's reported "issue with VA DTS EDFA".) The procedure is to move the AEC to a given position, AZ=90 EL=86 to ~ 90. (89.5, 89.6... ) using the mount panel, and then -after a command had been issued and the mount had reached its final position, he gets the encoder readouts using the Position Stream Client. This procedure was followed from 89 to 91 degrees EL and startting in AZ=90. The mounts moves in AZ and it went over 180 degrees, close to 270 degrees. A later test will track a point that moves over the zenith (transit.) By 17:10MST, Todd asked to move to a position 30 minutes before culmination at Declination 30 degrees but the antenna did not moved. Many unsuccessful tries were done and finally the mount panel of AEC displayed _Delay Getting State_ and a time later the ABM002/cppContainer was in the red (stop) mode. We tried to recover it using the "known" procedure, so we tried to put CONTROL in shutdown but CONTROL never reached the SHUTDOWN status and it 'frozen' on the PASS_1 status. So, the system had been fully restarted and the 6 real time machines had been restarted. In the previous procedure it was found the _maciContainer_ was still running on ABM002 and the machine hasn't unloaded the kernel modules after the container had been stopped. Once the system recovers the plan is to follow a point which is transiting over the antenna and track the readout positions against the commanded. For transit test we considere the ATF latitude is >~ 34 degrees and PSI was used to check the speed and acceleration of the mount in AZ are (6,12) [deg/s,deg/s^2] and (3,7) [deg/s,deg/s^2] in EL. [Roberto]
• 15:00MST, a black line ~ 0.5" width was visible on the Optical Telescope monitor. It was enough to adjust the V-Hold potentiometer since it was a problem with "vertical return lines". Not a proper issue, forget about it. I will consider 1 hour downtime given the failures (axis disagree and delay getting state, mostly because of the recovering time.) [Roberto]

• Activity Log: Active 13 hrs, Passive: 10 hrs, Unscheduled: hrs, : Downtime 1 hrs

January 12 Saturday

• ATF : 16:45 - 18:15 Emilio Barrios self training on ATF operation.
• CIPT: 08:00 - 16:45 Regression test done by Pablo Burgos & Jorge Sepulveda

• PSI software: Updated Central rack modules and PSI engine - details in JIRA
• PSI archiving: Evaluated issues with archiving update
• PSI M&C: Compared Central rack M&C ICD's with empirical results
• PSI planning: Developed method to handle expected LCU name changes at ATF and OSF - 10hrs total (Gene)

• Activity Log: - Active: 16 hrs (development, training), Passive: 8 hrs, Unscheduled: 0 hrs, Downtime: 0 hrs

January 11 Friday

• 16:15MST, Emilio and Roberto did 2 different exercises with the system. First, we did observed the effect of the pointing model on the movement of the antenna when even absolute coordinates (AZ, EL) are used. Without the pointing model the antenna reaches the requested positions even at 90 degrees EL (shutter was closed) but, as soon as we installed the pointing model, the error in AZ increases as the EL get closes to 90 degrees. Second, we did an exercise on the use of the Central LO Rack. We started checking system parameters just using the LabView^? gui's Baseband Total Power Detector. A problem was found for VA. The signal instead being a stable 'noise-like' signal around 1V, was a square-like oscillating but non periodical signal. It did not fixed using the PSI scripts 'rs sus' + 'rs csib104l'. Then we opened the Central LO Rack gui and tried to lock the WCA units. It worked on AEC but on VA it did not: the LLC on VA was lock but its status oscillated almost following the oscillation of the baseband signal. It was not possible to lock the VA WCA. _Jason Candelaria_ was contacted through the cell phone and he finally lock the system. Through the phone he reported problems at the FLOOG level. I understood (R.A.) he did a power strip of both the ANALOG and DIGITAL rack and this finally fixed the system. It might be important to comment a similar failure was observed a week ago when the VA-helium compressor was in maintenance. That time we thought the signal was oscillating because the electronics was not cold as usual but warm. So, very likely the system had been failing and it was not properly fixed before. It is better to put an eye on this as apparently had not happened before. All of this, anyway, was a good exercise for us, Emilio and Roberto. Temperatures where verified and system was inside its normal working temperature range. So it looks like a specific failure on the electronics. By 18:45 we left the ATF and the PSI diagnostic scripts were left running.
• Misc - 4pm-midnight - back on 5.0.1 software, run PSI archives for Gene (he needs 24 hours as continuous of a coverage as we can get). (Debra)
• CIPT 8:30-4:30 Testing of ASDM production under ALMA-5.0.2 still have issues with fundemental software, (COMP-1812 for instance) but are progressing. We were unable to start subscans on the Correlator, this problem is being investigated by the CORR team in CV. The system has been returned to ALMA-5.0.1-12 and the system is working as are the PSI scripts.
• Misc - midnight-8am - Still on 5.0.2 software in the morning, no PSI archive scripts can be run. Unused time = unscheduled (Debra)
• PSI/AEC Installing Tramps to prevent Mice attack (Roberto, Juan).
• PSI/VA/AEC: Manually checked VA and AEC cryo temps and pressures afternoon....ok. (.25-hours) Juan
• PSI/VA/AEC: Manually checked VA and AEC cryo temps and pressures first time in the morning....ok. (.25-hours) Juan
• PSI/VA: Cleaning Cryo oil stains on VA antenna, stains due oil leak we had on the compressor...ok. (2-hours) Juan, Roberto

• Activity Log: - Active: 7 hrs (development), Passive: 8 hrs, Unscheduled: 9 hrs, Downtime: hrs

January 10 Thursday

• Misc - Still on 5.0.2 software, no PSI archive scripts can be run. Unused time = unscheduled (Debra)
• PSI/BE - 5-7pm - Gene work remotely to collect monitor data for the EDFA's in the BE rack. Used 5.0.2 software which is in semi-usable state. Gene knew this and was still able to get what he needed. (Debra for Gene).
• 16:00MST, system still in CIPT hands, PSI not working. However, Paula Metzner had let us know that she does not need to run her test (on the DRX metaframes stability) any longer.
• 11:00MST, Juan Gallardo had been working in the AZimuth bearings greasing (checking drawings and docomentation plus visual inspection on site), but the access to the 16 nipples to lubricate bearings is not very clear. So, this is in 'pause' while waiting for a Pascal Martinez hint.
• 09:30MST, BE people reinstalled IF Processor modules in the analog racks of both antennas. A problem was found by Drew, a rare behavior of the pseudo front end. BE people found the main AC supply outlet was loose so they used duct tape to fix the cable to the floor and the outlet in place against the wall. Also, a DB( termination was disconnected. Then, the PFE had been reported working fine. Downtime ~ 1 hr.
• BE - 8am-noon - BE install IF processor module. (Hector Malagon, Michael Pursley). Details below:
  • Left AOC at 8:00 am, 9:00 am arrived at ATF.
  • By 9:20 am antennas were unlocked and parked in the maintenance position by the operators.
  • By 9:40 am IFP sn 103 and 104 were installed in the AEC antenna, and IFP sn 105 was installed in the Vertex antenna.
  • Attempts to verify IFPs operation were delayed due to computing software issues that prevented remote access to the antennas by Back End personnel. Eventually computing software was restored and all IFPs operation was verified, remotely by Drew Medlin. Through IFP verification in the AEC antenna it was determined that the PFE connected to IFP pol 1 was not working correctly. The operators returned the AEC antenna to maintenance position and we found that the PFE power supply cable was barely in the power socket and the CAN termination for the PFE was disconnected against the wall, a condition apparently the result of earlier maintenance, possibly a cryo rebuild. Duct tape was applied liberally to minimize future separation of power and CAN termination.
  • Left ATF at 10:30 am, 11:30 am arrived at AOC.
  • Notes:
    • If a callout list for Computing does not exist, maybe it should and be available for support when operators are not present at the ATF.
    • Extra care needs to be taken when working around the PFEs. The disconnected power and CAN terminator had to have been stepped on/kicked multiple times to achieve those results. Also, apparently there is no longer support for PFE CAN communications; is this a problem for the future diagnosis?
    • This activity was coordinated with Jeff Kerns, but some how was not put on the schedule. Better clarification on both Back End’s and Computing’s responsibilities for up coming activities should be taken care of early on. Back End was under the impression the modules were to be delivered to the AOC, while Computing was expecting Back End to pickup the modules.

• 07:50MST, DV01/cppContainer is down. It is possible to fix this or it will affect Lacasse and CIPT work? Answer, Ralph worked in this issue.
• PSI/VA: Manually checked VA and AEC cryo temps and pressures in the afternoon...OK (.25-hours) Juan
• PSI/VA: Manually checked VA and AEC cryo temps and pressures first time in the morning....ok. (.25-hours) Juan

• Activity Log: Active: 12 hrs, Passive: 0 hrs, Unscheduled: 11 hrs, Downtime: 1 hrs

January 9 Wednesday

• Misc - 5pm-midnight - no PSI scripts can run on 5.0.2 software - this is unscheduled time (Debra)
• BE - 4-5pm: Report from Hector Malagon, Michael Pursley: At the request of Jeff Kern, arrangements had been made to update IF Processor firmware to perform Ethernet data transfer verification. Sylas Ashton and Jeff had been working for some time on code tweaks and network tests at the ATF. Jeff and Bill Holmes had set Wednesday evening to perform the upgrade as a non-interference time with work at the ATF. Back End had agreed to work during the evening so that the IFP modules would be available the next morning. Detailed log of today's activities:
  • Left AOC at 3:00 pm, 4:00 pm arrived at ATF.
  • By 4:20 pm antennas were unlocked and parked in maintenance position by the operators.
  • By 4:40 pm IFP sn 103 and 105 had been removed, CAN jumpers installed, terminators installed on LO signals, and antennas returned to stow position.
  • Left ATF by 4:45 pm, 5:45 pm arrived at AOC.
  • From 5:45 pm to 8:35 pm IFP sn 103, 104, and 105 were opened up, the TPD board and the RabbitCore^? board were reprogrammed with the latest firmware 1.3v, modules were closed up, basic functionality tested in IFP test rack, and all cover screws installed following recommended torque sequence.
  • At this point Drew Medlin determined that he was unable to remotely move the antennas due to a computing software issue that prevented remote access to the antennas. We were unable to get a hold of anyone from computing that might be able to fix the problem. It was decided to wait until morning when operator support would be available at the ATF to return the IFPs.
• CIPT - 7am - 4pm - Correlator firmware update and work with 5.0.2 development (update did not happen, firmware bug found, troubleshooting proceeds, 5.0.2 is "buggy"). (Debra for CIPT)
• CIPT - 2-7am - unscheduled after rebuild (Debra for CIPT)
• CIPT - midnight-2am - finish software rebuild (Debra for CIPT)
• 13:25MST, As I understood it, Rich Lacasse had troubles with the new correlator firmware so, Jeff, Rafael and Steve had been spending time in to patch the system for later development.
• 08:15MST, Emilio and me had visited the AEC cabin as part of the 'training'. It had been observed that (assuming the antenna parked at AZ=66degrees, i.e. 'looking EAST) the left side channel for cables and other hardware, is a bit open in the section close to the cabin, thus exposing the cables to the weather. May be this requires some fixing/maintenance? This is more notorious when the antenna is pointing at ~ 90 degrees.
• PSI/AEC/VA: Software down so did a manual check on the cryo pressures, reserve tank pressures are 140psi on both antennas. (1hr) - Jack & Juan

• Activity Log: Active: 12 hrs (development, SW rebuild, BE), Passive: 0 hrs, Unscheduled: 5+7 hrs (after rebuild), Downtime: 0 hrs

January 8 Tuesday

• BE - 7-8am & 4-5pm - troubleshooting metaframe slips (Debra for Paula). Paula writes:
  • Right now I am gathering information. And I'm trying to get the information in an easy to read state. I was only able to get one DRX into the "unstable meta-frame" mode by resetting (this is the only time it seems to happen now - when it is the DRX causing the problem).
  • Because I have not seen much of this instability since I put in the last firmware update - I am suspecting that I have fixed the main culprit (namely a reset to the PLL). Although it could be that Gene is simply not there to report it (I don't think that's true - I can see how long the DRX has been running without any errors by a quick spot check.)
  • It could be this last DRX (x140) is either having another problem or I need to clean the fibers to it - DRX x140 has the lowest input power of all the DRXs by about half - although the transmitters are reporting the power output on all modules to be very close.
  • Still speculation at this point.
• 17:35MST, yesterday and today it had been observed that the PSI alarms are reporting ABM001 with no Hard tics but the system had the HARD tics. It looks like PSI is looking the wrong machine or something like that. Right now there is no "t01-abm"==ABM001 but "dv01-abm". In fact, sometimes PSI reports (in the start up, so is not possible to save the error message) problems at CCL/CVR level that eventually could explain the false error.
• 17:00MST, we had ended the operator cross training and we are leaving the system for CIPT new build. We have to report a problem when we used the _minicom_ and even if the data was right (we wanted to send the antenna to 45 degrees elevation) it reached the down limit (2 degrees) but minicom and PSI reported ~ 112 degrees. Brakes were released which was a good training for us and then we used minicom to clear faults and PSI to initialize the axis. A brief note, because the failure was in ELEVATION the initialization of AZIMUTH did not reported any problem but it was needed to clear faults in EKL and then it was possible to initialize this axis. Also, Paula Metzner called and asked to re run her PSI test script but by 17:05 I stopped in preparation for the CIPT new build (by Thomas Powers.)
• 16:30MST, a short test was done related to JIRA ticket COMP-1622. A difference of almost a minute had been found between the Local Sidereal Time given by the OMC and the LSt provided by a separate, 'well known', web site. Further testing is still needed al long Preben Grosbol reported this JT as 'fixed'. Or the fixing is not yet in the build in use in the ATF. As an extra check, the UTC in the OMC and in the website and system are the same, thus 'bounding' the problem to the LST only.
• Operators cross training. Moving the Antennas manually, shows the CORR, LO and real time machines, VLA Operation Control, etc
• ArrayTime^? (Thomas Juerges & Rodrigo Amestica, 4.00 hrs): Continued debugging. Rodrigo Amestica and I identified new problems in the synchronisation process of the clock in art-lo-1 with the GPS clock and the CRD. Rodrigo and I started to discuss possible solutions. We will continue tomorrow in Socorro, no access necessary for this.
• PSI/VA: ABM001 down, can't monitor anything. - Jack
• PSI/AEC: ABM002 is up and PSI scripts shows Cryo temps and pressures ok. - Jack
• PSI/VA: Manually checked VA cryo temps and pressures...ok. (.25-hours) Jack
• PSI/ATF: Went to the antenna barn and borrowed the forklift (wouldn't start!) to move the FE pallet truck from the container to the Certified Packing truck for shipment to Chile. (1.5-hours) Jack
• PSI/AEC: Moved manlift into place at the AEC antenna to do subreflector maint today at 10am. (.25-hours)Jack
• PSI/AEC: Its a cold morning (-4.5C) so I tried to move the AEC subreflector and it moved from (0,0,-4880) to (100,100,0) and back on the first command attempt. (.5-hours) Jack
• PSI/AEC: Lubricated the AEC subreflector and exercised. - (3-hours) Juan G., Jack
• PSI/ATF: Got the word from Rick M. that the 2nd holography system will not come to the ATF for acceptance so we can disassemble the holography tower, fiber, enclosure. But wait! Dick S. says we wait until April before doing anything with the tower. - Jack
• PSI/ATF: Certified picked up the shipment. - (1-hour) Jack, Juan G.
• PSI/ATF: Returned forklift to the antenna barn. (.5-hours) Jack
• PSI/AEC/VA: Rechecked both HE compressors for oil/helium leaks, looks good, and charged each reserve tank to 150PSI. (.5-hours) Jack

• Activity Log: - Active: 4.5 hrs (development, maint, tests) 5 hrs (training) 6 hrs (software rebuild) , Passive: 7 hrs, Unscheduled: 0 hrs, Downtime: 0.5 hrs

January 7 Monday

• 16:35MST, is snowing "heavy", both antennas parked and locked. PSI scripts running on snickers, we are now leaving.
• 16:12MST, tests of buttons CLOSE/OPEN on _mountPanel_. We had found these buttons work perfectly on VA but on AEC is not possible to OPEN the shutter because the _mountPanel_ shows the OPEN button is always in grey state. We had used _PSI_ to open the AEC shutter. Then we used the CLOSE button on the mountPanel and that button works. We had observed that when the shutter is closing, then the OPEN button changes to an OPERATIONAL dark status but, as soon the shutter is close, the CLOSE button remains DARK but the OPEN button backs to grey status thus being unavailable for a normal use.
• 11:00 MTS Operators training with Antennas. But, the weather is degrading and after noon it started snowing. Both antennas are in survival stow position. The practice had been just the start of the system and the use of tools from PSI (alarms). ABM001 had not hard tics. This was checked as real and then corrected but PSI still reports the same problem: might it be possible the alias _abm001_ is not being recognized since last friday it became _dv01-abm_ ?
• 09:30 MTS The only CORR Container found Up was CORR/ACC/JavaContainer all the other CORR containers were down and CORR subsystem was in ERROR. pstrip to ccc and cdp were executed. After this, the containers were restarted and CORR subsystem was sent to Operational. System is normal now. (Up and Operational) Same problem reported on Jan. the 5th.

• PSI/BE: No AEC LORR swap today as scheduled. - Bill B.

• PSI/AEC: Locate necessary lubricating tool at the VLA auto shop to do AEC subreflector maint tomorrow. (.5-hours) Jack
• PSI/AEC: Exercised subreflector and it responds. (.5-hours) Jack
• PSI/AEC/VA: Cryo temps and pressures look good this morning. - Jack
• PSI: FEC x10 VA 3 107.692 RES_TANK_PRESSURE
• PSI: FEC x10 AEC 4 92.186 RES_TANK_PRESSURE
• PSI: FED x14 VA 2 13.97 K GM_2ND_STAGE
• PSI: FED x14 AEC 2 13.50 K GM_2ND_STAGE

• Activity Log: - Active: 5.5 hrs (operator training), Passive: 18.5 hrs (high because close early due to weather), Unscheduled: 0 hrs, Downtime: 0 hrs

January 6 Sunday

• 10:00 MTS Regression test on going (Nicolas T., Nicolas B. and Rodrigo A)
• 09:00 MTS All System were found down and in ERROR. That's why Jack could not run PSI. Open runOMC panel wasn't send automaticaly the system Up. Only after a nukeDiskless could be possible to restart the system. Now it is Up and Operational
• PSI/ATF: CCL error this morning, can't run PSI scripts to monitor cryo temps and pressures. Could we fix this someday? Jack
• PSI/ATF: Replaced floor tiles in the control room. (.5-hours) Jack
• PSI/AEC: FE 2nd stage is an acceptable 18K and the reserve tank pressure looks good. (.5-hours) Jack
• PSI/ATF: FE overtemp document for Twiki. (.5-hours) Jack
• PSI/ATF: Continued scraping prototype nutator and packing instrument fixtures for shipment to Chile. (2-hours) Jack
• PSI/AEC/VA: FE's are at working temperatures and cryo pressures are good.
• PSI: FED x14 VA 2 14.27 K GM_2ND_STAGE
• PSI: FED x14 AEC 2 11.21 K GM_2ND_STAGE
• PSI: FEC x10 VA 3 108.547 RES_TANK_PRESSURE
• PSI: FEC x10 AEC 4 92.063 RES_TANK_PRESSURE

• Activity Log: - Active: 7 hrs (cryo rebuild, development), Passive: 8 hrs (back to normal after rebuild), Unscheduled: 0 hrs, Downtime: 9 hrs (cryo rebuild when nothing else scheduled)

January 5 Saturday

• 10:00 Regression test on going (Nicolas T., Nicolas B. and Rodrigo A)
• 09:00 MST. CORR/CDPNode and CORR/CCC Containers were found down and CORR subsystem in ERROR state. pstrip to ccc and cdp were executed. After this, the containers were restarted and CORR subsystem was sent to Operational. System is normal now. (Up and Operational)
• PSI/AEC: Removed leaky oil sight glass/flow gauge and replaced, started evacuating the HE compressor. (3-hours) Jack
• PSI/AEC: Removed subreflector covers and inspected subreflector cables, gearbox, electronics. () Jack & Juan G.
• PSI/AEC: Using PSI rs move commands, we were able to command the subreflector to several positions and then monitor that it had moved to the commanded position. Outdoor ambient is not so cold so maybe this has some influence on being able to move the subreflector. We have the correct lubricant to perform maintenance on the subreflector but lack the proper tool to inject the grease so we will need to look for and order the proper tool before continuing with the maintenance. (2-hours) Jack & Juan G.
• PSI/VA: Reinspected HE compressor for leaks. The VA cryo system is at 4K this morning. Charged reserve tank to 150psi. (1-hour) Jack
• PSI/AEC: Charged HE compressor and reconnected helium lines and restarted cool down. Too windy to leak check today. Will leak check tomorrow. (1-hour) Jack
• PSI/ATF: Began scraping out prototype nutator and will save motors for Nick Emerson. (1-hour) Jack

• Activity Log: - Active: 7 hrs (cryo rebuild, development), Passive: 0 hrs, Unscheduled: 7 hrs, Downtime: 10 hrs (cryo rebuild when nothing else scheduled).

January 4 Friday

• 15:40MST, the system had been released for the use of Jack and Juan who had been working on the AEC antenna. They had taken out the first plate which protects the rear access to the sub reflector, in preparation of the greasing work to be done tomorrow saturday.
• 13:10MST, system had been switched back to 5.0.1.10 and now is OPERATIONAL. Operator tests on AEC starting at 13:20MST. Pointing on Mercury and Jupiter, FivePoint^? and use of Object Explorer.
• 11:30MST, Jeff and Debra had been working around the requirements for the "alarmPanel", mainly in the issue of alarm status and their colors.
• 07:50MST, OMC under VNC session is not running and all PSI scripts are down. There is no info on the emails and it looks like it was shutdown in order so, guessing someone is doing something we'll wait (I had not found nobody at the tube also.) As an extra observation, someone had remotely shutdown the VNC session. 08:30MST, Nicolas had been working on _ACS : ACS-7.0.0 Build : HEAD-2008-01-03_ thus explaining the observed behaviour of the system. By 08:50MST, all is OPERATIONAL but TMCDB is in UNKNOWN state.n Unsolved but not important.
• PSI/VA: Continued with HE compress