May 31 Thursday - Holography equipment removed from ATF!

• ATF AEC: Holography mini rack and receiver were removed from the AEC antenna so that they may be prepared for shipment to chile - Jack, Juan, Drew, Roberto
• ATF HOLO: The HOLO transmitter as removed from the holography tower today in preparation for its shipment to chile - Jack, Peter, Roberto
• ATF VA: Radiometric pointing was resumed using Mars and Venus. Mars was not detected, Venus was. Spiral program and mountGUI/mount need to be corrected or compensated before data can be collected - Gene

May 30 Wednesday

• COMP AEC: Computing is working with AEC software testing from the AOC - Ralph
• BE VA: BE came to the ATF to return LO2 s/n 2 to the VA after reprogramming from a memory corruption issue - Phil, Juan S., Drew
• PSI VA: Radiometric pointing efforts were taken today using the VA - Gene, Jeff, Drew
• PSI VA: LORR lost lock after BE work's analog rack power cycle. Optical power of the 1532 nm CRD signal was again measured: -6.60 dBm, still within theoretical locking range - Drew, Peter

May 29 Tuesday

• PSI ATF: Certified Packing will pickup OPT for shipment to Chile. - Jack
• PSI ATF: Gene is fighting with software (and losing) this morning. - Gene
• PSI VA: VA encoder re-calibration/test - Peter, Roberto, Juan, Drew
• ATF OPT: VA OPT has been picked up by the ABQ packing company - Drew, Jack, Packing crew
• ATF OPT: AEC OPT restored to functionality by working on the shutters. Solar filter motor shaft had become loose. Setscrews were tightened and some minor changes to the cam shaft sensors were made as well - Drew, Jack, Peter

May 28 Monday Memorial Day, NRAO Holiday

• PSI AEC: Turned off holography transmitter and then receiver for Gene to see signal beacon. - Jack
• PSI AEC: Repositioned AEC antenna for best beacon signal. - Jack, Peter R.
• PSI/AIV: Packed up remaining OPT items. OPT ready to ship! - Jack, Peter R.
• PSI Vertex/AEC: Both antennas pointed at the beacon overnight for tests. - Gene
• PSI Vertex: Replaced two antenna panels and grounding system. - Jack, Peter R.
• PSI/AIV: Grilled steak, chicken, and pork (don't want to leave out a meat group) to celebrate Memorial Day! - Jack, Peter R.

May 27 Sunday

• PSI AEC: Reinstalled absorber in feed area for holography. - Jack
• PSI AEC: Reset FE power supply and checked WMA voltages. - Jack
• SCI AEC: Darrel is contiuing with holography testing. - Darrel
• PSI Vertex: Removed OPT, mini-rack, power supply. - Jack, Peter, Juan G.
• PSI AEC/Vertex: Moved both antennas to the beacon, removed absorber from AEC, and opened the shutters. - Jack, Peter
• PSI ATF: Began assembling all OPT equipment for shipment to Chile. - Jack, Peter

May 26 Saturday

• PSI Vertex: FE Lakeshore temp monitor will be removed and analyzed why it doesn't work. - Jack
• PSI Vertex: FE Lakeshore problem solved by resetting dewar monitor module. - Jack
• PSI Vertex: Now there is a FE JT flow readback problem, will investigate. - Jack
• SCI AEC: Darrel is continuing with holography. - Darrel, Roberto
• PSI Vertex: Removed 2 backup structure panels to access OPT tomorrow. - Jack
• PSI CLO: Gene is reporting LORR and LS lock problems on AEC, a visual inspection shows all of the modules powered on. - Jack
• PSI Vertex: Rounded up some temp brackets to mount the ACU to the rx cabin floor after removing the OPT mini-rack. - Jack

May 25 Friday

• BE on site troubleshooting Vertex rack module issues. - Stian, Phil
• PSI AEC: Looked at EL balance currents since OPT mini-rack was moved out of rx cabin. - Jack
• SCI AEC: Darrel continues with Holography. - Darrel
• PSI Vertex: Installed MMF from AZ platform Ethernet switch to the OPT Ethernet switch. - Jack, Ralph
• PSI Vertex: Resynched GPS receiver in the OPT mini-rack. - Jack, Ralph
• ATF: Tornado spotted just north of the Alamo Reservation - Tornado
• PSI Vertex: OPT mini-rack ready for testing tonight if weather is good. - Tod
• PSI OPT: Packed up OPT spare parts for ship to Chile next week. - Jack
• PSI AEC: Holography rx needed 48vdc reset. - Jack
• SCI AEC: Software problems, very little progress on Holography. - Tod, Darrel, Roberto, Manuel, Nico

May 24 Thursday

• VLA power outage for UPS installation, network down.
• PSI Vertex: Problems with OPT Ethernet communications. - Jack
• PSI Vertex: FE Lakeshore temp monitor display is blank. - Jack
• SCI AEC: Darrel continues with Holography. - Darrel
• SCI Vertex: Tod Hunter continues with OPT. - Tod

May 23 Wednesday

• VLA power outage for UPS installation, network down.
• PSI Vertex: Replaced compressor oil filter. - Jack
• PSI AEC: Bypassed two shorted AZ drive motors. - Jack, Mike R.
• PSI AEC/Vertex: Moved OPT mini-rack from AEC to Vertex. - Jack, Juan G., Roberto, Drew
• PSI Holography: Started inventory list for shippment to Chile. - Jack, Juan G.
• PSI ATF: Certified Packing Rep on site to look at shipments to Chile. - Drew
• SCI AEC: Darrel continues with Holography. - Darrel
• SCI Vertex: Tod Hunter continues with OPT. - Tod

May 22 Tuesday

• ATF AEC: Receiver cabin temperatures are still hot: even after leaving the doors open all nightf, the omega reads 29 C. temperatures are slowly falling, but they are still well above the 22 C cutoff for BE rack operation. The doors have been closed, and all BE equipment is powered off safely. Holography will now continue until more efforts may be put towards the HVAC issue - Drew, Darrel, Jeff, Hank
• HOLO AEC: Holography maps continue - Darrel, Jeff, Todd
• ATF VA: LO2 40 checked for output signals: 14 GHz @ 3dBmV - Juan Pablo
• PSI AEC: HVAC back up and operational. - Jack
• Certified Packing (ABQ) rep came to the ATF for a "look see" before shipping Holography and OPT to Chile. - Drew
• PSI Vertex: Cryo compressor is off, receiver is warm. Same problem as the AEC compressor oil filter clogged. I will replace at first opportunity. - Jack

May 21 Monday

• PSI AEC: Holography insulation foam was replaced in the shutter area of the AEC antenna - Robert, Juan
• HOLO AEC: Holography maps are started - Darrel
• PSI ATF: Master Laser work continues - Sarmad
• ATF AEC: The AC was found to not be properly working, and the temperatures in the AEC receiver cabin reached high temperatures. Efforts will be made to restart the AC system. Part of the AC system is in the hot receiver cabin, and must be cooled to lower pressure before the AC may be restarted . . . that means the receiver cabin must cool down before we can cool it down (yes i know what i typed) - Juan Pablo, Jack, Darrel, Manuel, Rodrigo, Juan

May 20 Sunday

May 19 Saturday

May 18 Friday Heavy rains in the late afternoon

• OPT AEC: OPT shutter closed and the solar filter did not. Closer inspection shows no clear mechanical interference between the two. The shutter was manually opened, and the solar filter closed itself automatically. Approaching thunderstorms warranted the closure of the solar filter and shutter, so power was turned off in the receiver cabin. Again, only the shutter closed. The solar filter was left open due to close lightning prohibiting manlift use - Drew, Juan, Alison, Jeff
• ATF AEC: The lockout knob on the base of the AEC antenna was found to be very loose. It pulls off the antenna with little effort, causing lockouts to be not possible, as the exposed metal shaft could be turned with pliers, or a nut driver, powering the drive motors. A setscrew was found on the knob, so the know was replaced and tightened appropriately. This should, for the time being, restore more or less proper lockout ability - Drew
• ATF VA: OPT work has moved to the VA for the next week at least - Jeff

May 17 Thursday

• OPT: OPT shutter fails to respond to commands to open or close. Shutter reports closed and open when using PSI to command the shutter, while it only stays closed. Shutter was manually opened, and the solar filter closed itself each time it was opened manually. After some software efforts, both the shutter and solar filter remained open when manually moved - Drew, Juan, Gene, Alison

• Right click the images and 'view image' to see larger sizes - Drew
• OPT from the side . . . shows the inclination angle:
  

• OPT with shutter closed:
  

• OPT solar filter . . . with dirt . . . and a handprint!?:
  

• OPT objective . . . if you love telescopes this one will make you want to cry:
  

• OPT lower left, more clearance than before (see below):
  

• OPT upper right, less clearance than before, but still not touching:
  

• OPT with open shutter, side view:
  

• OPT shutter motor shaft area:
  

• OPT solar filter motor shaft area:
  

• OPT solar filter motor shaft area . . . green surface not covered by cam:
  

• ATF VA: BE work on IFDC issues in the VA were addressed today - Stian, Nick, Gene, Drew
• ATF AEC: BE and PSI addressed power supply issues in the AEC antenna. The PSD is being replaced as no solution to the current PSD power up poroblem could be found. PSA is also being swapped from the VA - Stian, Nick
• PSI VA: PSI will re-calibrate the VA El encoder to attempt to solve the 10 arcsecond jumps - Peter, Drew, Juan

May 16 Wednesday

• OPT: OPT work continues with the AEC antenna. Problems with the Az drive shutdown are still in the way - Alison, Roberto, Manuel, Gene, Drew
• PSI VA: Fixing IF MUX box gain and ripple - Robert, Rodrigo
• ATF: Dan Mertley borrowed the TEREX manlift for RFI work on the RFI tower for the better part of the day.
• ATF: Installed two cable trays in the AEC trailer main room. One is on the bookshelf back next to the workbench, the other is in front of the bookshelf near the door - Drew
• ATF: Installed the new 8 port switch in the AEC trailer conference room - Drew
• ATF AEC: AEC digital rack DC power does not come on. Fuse just above the switch looks fine - Drew, Gene
• OPT AEC: OPT shims were removed, 3mm worth from each of the bottom bolts - Drew, Juan, Manuel
• OPT AEC: OPT shims were found to cause a shutter error: shutter hits the tube for OPT and remains in an unknown state software-wise. Drew came back to the ATF at ~ 9:15 to replace 1mm shims on each of the lower bolts of the OPT. Shutter appears to function properly now - Drew, Manuel, Debra, Mauricio

May 15 Tuesday

• PSI Vertex: Swapped encoder EEU's between AZ and EL for more analysis. - Jack, Peter Napier
• SCI: Attempting OPT pointing but skies are somewhat cloudy. It will be good practice. - Angela, Roberto
• PSI: ML put back on UPS power - Drew, Jack
• PSI: AVATA is now on a portable UPS along with an external hard drive and monitor - Drew
• PSI CLO: ML troubleshooting with Jason Castro and Gene, no progress. Jack
• PSI Vertex: Swapped EEU's back to their original location and did some more tests. - Jack, Peter N.

May 14 Monday

• PSI AEC: Gene and I performed the tests on the AEC AZ drive motors for Pascal and Gene sent the results. - Jack, Gene
• PSI Vertex: Peter Napier and I performed encoder tests. We swapped encoder inputs at the ACU and the problems stayed in EL.
• Computing/SCI: Debra Shepard, Angela, and others debugged software and trained for OPT.
• PSI CLO: Tried to locate configuration files on the ML computer without success. - Jack

May 13 Sunday

• PSI AEC: Disconnected cryo lines from compressor and connected evacuation lines and now waiting for an opportunity to repair. - Jack
• SCI AEC: Darrel is attempting to get holography maps before he leaves the ATF at mid-day. - Darrel
• SCI AEC: Darrel was able to get a couple of maps before turning the antenna over to OPT. - Darrel
• PSI AEC: Numerous problems getting OPT tests. Scheduling block problems, offset problems, possible sidereal time issues. - Jack
• Computing: Weather turning cloudy, at 3:45 Roberto, Manuel, Mauricio are leaving site. - Jack
• PSI AEC: I'll take this opportunity to repair the AEC helium compressor. - Jack
• PSI Vertex: AZ is also showing the same symptoms as EL servo jumps and thumps. - Jack
• PSI Vertex: Installed (3) zero-degree markers in the ped room to recover to 0 degrees if needed tomorrow after tests. - Jack
• PSI AEC: Completed helium compressor repair and its back on line cooling down the FE. - Jack
• PSI AEC: The AZ drive not ready problem appeared twice today. Both were recovered. - Jack

May 12 Saturday

• PSI AEC: Reinstalled loose absorber in the feed window. - Jack
• PSI AEC: Mount GUI problems, AEC antenna won't move. - Jack, Darrel
• Computing: Ralph analyzed problem with Mount GUI and recommened complete restart of software to recover. - Ralph, Roberto
• PSI CLO: Troubleshooting ML computer and sending into to Traxion and Jason Castro. - Jack
• PSI AEC: Received compressor oil filter replacement and will change out at first opportunity. - Jack
• Comp/Sci: Holography maps continue. - Darrel, Roberto, Manuel, Mauricio
• PSI Vertex: Checked out AZ/EL encoder cabling before Monday's encoder tests. - Jack
• PSI AEC: AZ drive crapped out again in the middle of a map (resetting AZ drive) . Tightened terminal block connections and wiggled relays in their sockets, checked opto relays, rebooted ACU and recovered operation. - Jack
• PM weather: local thunder storms and rain.
• PSI AEC: AZ drive crapped out again but recovered after an ACU reboot. Holography maps continue. No OPT tonight due to clouds. - Jack

May 11 Friday

• PSI AEC: FE shutdown to replace clogged compressor oil filter. - Jack
• PSI CLO: Continuing with ML computer troubleshooting. - Jack, Jason Castro
• PSI VA: IFDC S/N 3 was removed from the Pol0 (x29) slot in the VA, and IFDC S/N 2 was inserted - Drew
• PSI AEC: Swapped OPT/ACU cables to Holography mini-rack for Jeff K. and Darrel. - Jack
• PSI AEC: Adjusted OPT for Jeff Mangum. - Jack
• PSI CLO: More info from ML vendor, more tests. - Jack
• PSI AEC: Swapped OPT/ACU cables to OPT but clouds interferred with finding a star. - Jack, Roberto, Manuel, Jeff K.
• PSI AEC: Swapped OPT/ACU cables back to Holography and started taking more maps. - Roberto, Jeff K., Darrel

May 10 Thursday

• PSI BE: Stian and Nick on site doing temp data logger stuff. - Stian, Nick
• PSI CLO: Spoke to Jason Castro about ML computer problems and received new tests to perform. - Jack
• PSI AEC: FE compressor shut down on high temp, need to order filters today and replace asap. - Jack
• PSI AEC: Spoke to Jeff Mangum about OPT adjustments and will pass on to Ralph. - Jack, Jeff M.
• PSI CLO: Continued troubleshooting with Jason Castro and gave him results. - Jack, Jason C.
• PSI ATF: The network appears to be back up. - Jack
• Computing: Software debugging, holography maps. - Ralph, Jorge
• PSI FE: VA 4-12 vs 14-22 IF FE Tip tests - Peter, Gene, Robert R.
• PSI: System analysis - Drew details?
• Computing: Switched from holography to OPT and checked OPT alignment. Tonight OPT pointing. - Jack, Drew, Ralph, Roberto, Jorge
• PSI CLO: ML had lost lock probably due to an extended power outage. ML was reclocked manually with 0.175V fluorescence - Gene
  E2000 FO connector on ML rack was inserted but found to not be engaged - Jack
  LS locked for 80Ghz 1stLO - Gene

May 09 Wednesday

• VLA power disrupted until mid-day. Ralph had problems getting the server rack computers back up. - Ralph
• PSI CLO: Constructed a test circuit for Jason Castro and tested the ML problem, sent results to Jason. - Jack
• PSI ATF: Jody Bolyard on site to do safety analysis of Holography and OPT systems before shipping to Chile. - Jack, Jody, James S.
• PSI ATF: Mike Rodriguez on site checking on documentation and shipping of holography and OPT to Chile. - Jack

May 08 Tuesday

• PSI: Manlift rental picked up by United Rentals this am. Gas leak drained tank. Had to make arrangements with Lew Serna for gas and can - Gene

May 07 Monday - Clear and Calm

• HOLO AEC: Holography maps - Robert L.
• PSI AEC: Tests run for AEC ACU errors experienced recently - Pascal, Gene, Peter
• PSI AEC: Absorption foam was reseated in the AEC antenna for holography purposes - Drew

• AEC HOLO foam has fallen down:
  

• AEC HOLO foam put back:
  

• AEC HOLO foam near the HOLO receiver:
  

• PSI AEC: Photographs taken of the AEC OPT using the manlift - Drew
• AEC OPT:
  

• AEC OPT:
  

• AEC OPT:
  

• AEC OPT:
  
• PSI ATF: The rental manlift has been observed to have a fuel leak, evident by the lack of fuel in the tank, and the large wet spot under the manlift which smells of fuel - Roberto, Drew, Manuel
• OPT: Software issues were addressed by computing - Ralph, Jorge
• ATF: Due to a power outage for UPS work in the VLA building, all ALMA devices there are being shut down starting at 4:30 - Drew, Sarmad

May 04 Friday

• Partly Cloudy, low winds in the morning, high winds in the afternoon
  

• PSI AEC: When slewing AEC back and forth repeatedly in Az the Az axis shuts down within 10 minutes after start of slewing. If the slew tests are repeated in succession the time to shutdown decreases to less than 3 seconds. The Az shutdown is usually recoverable after issuing CLEAR_FAULTS command via AMB software. So far no error has been detected via M&C. Hampering this effort the ICD received at AEC Training course was found to be incomplete and the ICD#9 received this week has RCA descriptions not included in the ACU firmware at ATF. The above shutdowns occur when Az motor temperatures report above 90/70C. Cause is unknown though overheating is suspected.
  
  In about 1 out of 8 shutdowns the ACU must be reset either by issuing RESET_ACU_CMD via AMB or by pressing ACU reset button. This seems to be independent of Az motor temperature. The ACU shutdown is sometimes coincident with a cluster of system software errors. Cause is suspected to be CAN communication failure though effort is hampered by broken Openbrick.
  
  The antenna was observed once to be slewing after shutdown. Either the breaks were left disengaged and the wind moved the antenna or the AEC ignored trajectory command destination and continued to slew.
  
  While slewing in Az sometimes the AEC shudders. This can be heard inside the control trailer. The shuddering may be associated with overshooting as observed by Darrel and Robert. Also see antenna data previously sent. The ACU vibrates when slewing in Az or El regardless of speed and acceleration settings.
  
  Openbrick fails to boot.
  
  Issues observed last week: Two times after shutdown the Az slave driver reported G,G,G,R,G => over current - Gene
  
  
• HOLO AEC: Holography maps and troubleshooting of the AEC antenna - Jeff, Drew, Darrel, Robert
• HOLO AEC: The CAN cables connecting the HOLO Mini-rack to the Optical Telescope and the Holography Rx failed. Retaped and strain relief installed, later, the two pieces were more properly connected with bolts - Jeff
• ATF: High winds have caused the antennas to be stowed in their survival stow positions -
• ATF: A power glitch lasting only a second or two caused the UPS units in the control trailer to come on at 14:55 MST -

May 03 Thursday - Clear. The Return of the TEREX Manlift.

• HOLO AEC: Holography maps made with slow and fast speeds - Jeff, Roberto, Manuel, Darrel
• PSI VA: Tip tests conducted for stability testing - Gene, Peter
• ATF: Manlift testing after repair completion. Terex manlift brought back to the ATF - Juan, VLA autoshop crew
• OPT AEC: OPT work to see if the OPT shims were placed correctly to adjust the OPT axis closer to the radiometric axis of the AEC antenna - Jeff
• PSI AEC: This afternoon after a few hours of no operation the ACU was slewed 2 degrees back and forth in azimuth. After about three minutes the Az drive shutdown. The Az slave drive was reporting G,G,G,R,G again indicating an overcurrent condition had occurred. The ACU was hardware reset to recover but after the reset the Az axis failed to leave shutdown mode. El axis was okay. The ACU was soft reset but the Az still failed to leave shutdown mode. The Az slave was found to again be reporting an overcurrent error. The relays were banged with a screw driver then the ACU was hard reset two more times. After this the Az axis recovered and Darrel and Robert were able to continue with Holography.

• PSI AEC: The AEC was confirmed to be locked out with padlock this morning but ACU reports azimuth axis is in encoder mode and brake is disengaged. I tried commanding AEC to move via AMB but AEC did not move. I tried pushing antenna but brakes seemed to be engaged. If so then there is a disagreement between ACU and actual AEC status. If not then there is a serious safety issue. Either way we should require everyone to always push E-stop next to padlock when locking out AEC - Gene
  
  This e-stop failed mechanically about four weeks ago. Juan G. fixed the switch but now it disengages in revers of arrows. Once we start exercising this e-stop we may encounter more problems.
  

• ATF AEC: A crow landed on the top of the dish while the antenna was pointed at the holography tower. Attempts were made to move the antenna to shake the bird off, though it flew away of it's own accord before the antenna coudl be made to move.

May 02 Wednesdy - Thunderstorms and hail, great weather for holography

• ATF: Much effort today was put into finding issues with the AEC antenna ACU failures - Gene, Drew, Jeff, Manuel, Roberto, Darrel
• Manlift repair work - Juan

May 01 Tuesday - Partly to Mostly Cloudy in the A.M., thundershowers started just before 2, thundershowers the rest of the afternoon

• ATF AEC: Work continues on the AEC ACU shutdown issues - Gene, Jeff, Others
• ATF AEC: The brakes were released so Drew could push the AEC antenna around in AZ to test for a possible brake issue. Then several people were involved to push the AZ axis around to see if there was a noticeable difference in force needed from previous manual movements. Results: the brakes seem ok - Drew, Gene, Scott, Manuel, Roberto, Sarmad