Download PDF Fall Issue of The Bridge on Space Exploration September 1, 2021 Volume 51 Issue 3 Close collaboration between engineering and science has enabled marvels of space exploration over decades. Eight exemplary missions are described in this issue, conveying the excitement, challenges, and breakthroughs involved in efforts to better understand the wonders and mysteries of this solar system. Servicing the Hubble Space Telescope: A Partnership of Engineering and Science Wednesday, September 15, 2021 Author: John M. Grunsfeld Hubble is a monumental scientific success, a marvel of engineering, and an exemplar of teamwork. Operating in Earth’s orbit for 31 years, the Hubble Space Telescope (HST; figure 1) is perhaps the most productive scientific instrument ever created. Its chronicles are about more than a telescope in space. They are stories of imagination and dreams, of struggles, failures, and triumphs, sometimes against the odds. Background Since the start of its mission in 1990, Hubble has made more than 1 million observations resulting in over 18,000 scientific papers. Tens of thousands of astronomers around the world use Hubble, and many more mine its vast open-data archive. The images and data gleaned from the observations made by this unique observatory have provided clues to some of the most fundamental questions in science. Astronomers using Hubble have measured the age of the universe, revealed the mysterious “dark energy” behind acceleration of the expansion of the universe, seen back 13 billion years to study some of the earliest galaxies, proved the existence of black holes, peered into stellar “nurseries” where young stars and solar systems are being created, yielded remarkable insights into other planetary systems (exoplanets), and watched comets impacting Jupiter. Pioneers and Top-Notch Contributors It is often said that the work on the Hubble observatory represents the best marriage of engineering, science, and human spaceflight. Hubble is the premier science spacecraft in the NASA fleet because of the remarkable vision of the top-notch engineers who designed it to be serviceable and the technicians, astronauts, and flight controllers who have kept it operating for decades in the face of extreme challenges. It is impossible to name all the key players that enabled this awe-inspiring observatory, but no accounting of Hubble would be complete without saluting the contributions of Riccardo Giacconi (2008), the first director of the Space Telescope Science Institute, which is responsible for operating the science program and supporting astronomers in the use of Hubble. During development, the engineers who designed the observatory had to plan for interfaces with optical tolerances for instruments that could be serviced by spacewalking astronauts working in a vacuum (Smith 1989). To help enable reliable operations, Hubble was also designed with redundancy of critical systems and cross-strapping capability, which, combined with the servicing, have enabled the observatory to keep operating and remain state-of-the-art. One innovator who stands out as the sine qua non engineer, whose genius and creative drive ensured the success of the missions and who is credited for inventing space servicing, is Frank Cepollina. The team supporting Hubble represents nearly all the NASA centers, especially the Goddard Space Flight Center (GSFC), Marshall Space Flight Center, Kennedy Space Center, Jet Propulsion Laboratory, and Johnson Space Center (JSC), as well as the European Space Agency, industry, and academia. Over the course of five space shuttle missions, engineers, technicians, operators, and astronauts working together have kept Hubble at the leading edge of scientific productivity (Grunsfeld 2014). Table 1 outlines the work performed on those missions, plus the mission that launched Hubble into orbit. Examples of Critical Teamwork to Meet Challenges Hubble’s start was not so auspicious, and is itself a compelling engineering and science story. Soon after the telescope was deployed into orbit on April 24, 1990, an infinitesimal flaw was detected in the 2.4 m diameter primary mirror: it was too flat by approximately 2.2 μm at the edges, causing spherical aberration. The first pictures transmitted to Earth were a huge disappointment to the scientific community and to NASA (Gainor 2020). NASA rallied some of the nation’s top engineers and optical scientists to craft a repair with mirrors and lenses to correct Hubble’s vision. These would be inserted as a separate “set of contact lenses” in Hubble’s optical path, as well as new lenses built into the new camera to be installed during the first servicing mission. The “surgeons” tasked to install the new optics to restore Hubble’s vision were the astronauts. I had the privilege to work with the great HST team and fly on three space shuttle missions, earning the title “the Hubble repairman.” In the following sections I describe a few illustrative examples of the innovative technology and engineering that went into the spacewalking efforts to upgrade and repair the telescope on orbit. Doing Precision Repairs in Hockey Gloves Even with brilliant engineering and planning, keeping Hubble operating required avionics and system repairs that were never intended to be done in space by astronauts in bulky spacesuits with thick gloves better suited for playing hockey than repairing electronics. Fortunately, one of the defining skills of the human species is the ability to engineer tools and processes to meet a challenge. By creating new tools and techniques, the team overcame the difficulties posed by operating in a spacesuit, enabling work to be done in space at the same level of detail as on the ground by skilled technicians. The typical HST mission flew with hundreds of custom tools. My experience is replete with examples of the crew, engineers, technicians, machinists, and spacewalking trainers working together to develop new tools. For my first Hubble servicing mission in 1999 we had to develop a miniature torque wrench for use on subminiature assembly connectors, which, with their fine threads and 7–9 in-oz torque requirement, were a challenge to remove and install in the bulky gloves. On the final servicing mission in 2009, we had to remove the cover plates of instruments to remove failed circuit boards, a task never attempted in space. The team developed many new tools to cut aluminum electromagnetic interference grids, remove dozens of #4 Torq-set and #4 socket head cap screws, and replace circuit cards without cutting the space suit gloves on their sharp G3 glass laminate. Without all the creative custom tools and plenty of practice, Hubble would not be performing in orbit today. Training for a Year to Replace the Main Power Control Unit On the fourth servicing mission in 2002 we had the unenviable task of replacing the main power control unit (PCU). It contained all the relays to power Hubble and had 36 large circular cannon-style connectors with extremely stiff, difficult-to-handle heavy power and signal cables running down its left side. In testing we found that existing connector tools were ill suited for the PCU repair task. Through many iterations, including testing on a high-fidelity simulator, the GSFC tool development team and I developed a new tool called the high-torque connector. We went through a wide parameter space of size, jaw geometry, grip pad durometer, plier pivot location, restoring spring force, and tether location (all tools have to be tethered or they will float away). We had the benefit of the then new technology of additive manufacturing to be able to make rapid prototypes for testing in the neutral buoyancy test facility at JSC where I used the tool in the space suit on an HST mockup underwater. FIGURE 2 The power control unit mockup used in training for the fourth servicing mission in 2002, and (inset) the high-torque connector tool. Image credit: NASA. The PCU tool may look simple (figure 2), but it is a highly optimized solution for a critical task that recruited multiple engineering disciplines and human factors to develop. Often the most elegant engineering solutions look simple, hiding the complexity of the development process. The PCU repair task exemplified the high-performance challenge of repairing Hubble in orbit. Had we not tried to repair the failing PCU, the whole observatory would have gone dark because of a faulty main power buss bar. This task was of such importance, high risk, and difficulty that the final decision to do the repair was made by NASA administrator Dan Goldin. Nearly every evening for a year I worked on a high-fidelity mockup of the PCU, meticulously removing all 36 connections and then reinstalling them using the new high-torque tool and spacewalking gloves. I wanted to know every connector personally, all the features of the keying of the circular connectors, the cable routing, even the pins and sockets of each connector (which I would have to inspect in orbit). Making matters even more difficult, the door on the telescope interfered with my space suit helmet, blocking my view and preventing me from having stereo vision of the connectors. I had to learn to attach the connectors by feel and a view from my left eye only. On game day my spacewalking partner Rick Linnehan and I finished the task in a total spacewalking time of 6 hours 48 minutes, much less than predicted. I give credit to the special tools the team had developed, the high-fidelity simulators, the incredible depth of documentation including as-built drawings and close-out photos showing the actual configuration of hardware in the telescope, attention to every detail, and lots of training. These are the best practices that led to the success of all the servicing missions. Overcoming an Intractable Bolt to Repair the Camera While the tools were contributory factors to the success of the servicing missions, the most important factor is the broad knowledge and extraordinary teamwork exhibited by the Hubble team. On the final space shuttle mission to Hubble in 2009, the highest-priority objective was the replacement of the Wide Field/Planetary Camera 2 (WFPC-2) with the new state-of-the-art Wide Field Camera 3 (WFC-3). The WFPC-2 had been installed on the first servicing mission in 1993 and is credited with saving Hubble as its internal optics corrected for the primary mirror’s spherical aberration (Zimmerman 2008). The WFC-3 incorporated the same optical correction but also a much larger format image detector with lower noise and greater sensitivity in the UV and visible wavelength range, as well as a new near infrared image sensor, enabling the instrument to see deeper into the universe and to peer through regions of the cosmos with micron-sized dust to reveal mysteries within. The replacement of the WFPC-2 was scheduled on the first spacewalk of the mission. We used mockups of the instruments to practice in the large Neutral Buoyancy Laboratory pool in Houston to train removal and installation in the telescope. At the GSFC we worked with the actual flight instrument in a high-fidelity mechanical simulator of the telescope. Through training and study of the engineering documentation, I learned the details of the mechanisms, electrical connections, and important features to make sure we could properly install the new instrument. As both an astrophysicist and an instrument builder I was particularly excited to install this new camera that would help astronomers unravel the mysteries of the universe. With a big grin on my face (I love spacewalking) I set out with Drew Feustel to replace the WFPC-2 instrument (figure 3). FIGURE 3 Astronauts John Grunsfeld (red stripe on backpack) and Drew Feustel attempting to remove the Wide Field/Planetary Camera 2 during the fifth Hubble servicing mission, STS-125, in 2009. Grunsfeld awaits instructions while Feustel configures the ratchet wrench. Image credit: NASA. We immediately ran into a big snag. The primary bolt holding the WFPC-2 wouldn’t budge. Our ratchet wrench had a torque limiter, essentially a stack of Belleville washers and springs with a maximum of 38 ft-lbs. I retrieved a backup torque limiter with a maximum of 45 ft-lbs but still no luck. In the cabin the rest of the crew was looking at the contingency checklist (the “what to do when things go wrong” book) and consulting with the extended engineering team in mission control for guidance on how to proceed. Floating outside the telescope I recalled that the designers had incorporated a thinned section in the shaft that runs from the bolt head at the front to the threads at the back. This ensured that if the bolt failed it would fracture in the middle rather than the high-stress area at the start of the threads locking it in. If the shaft fractured in the thinned section, the old instrument would be locked in place and the new instrument would return with us to Earth. I also remembered that the shaft would fracture around 57 ft-lbs. The grin on my face was gone. Mission control gave us the direction to try to remove the old camera with no torque-limiting protection. Either the shaft would break in the middle or the bolt would loosen properly. Drew gave a slow and steady push on the wrench and suddenly it turned, although we didn’t know whether it had broken or come loose. Using the pistol grip tool (PGT), a microprocessor-controlled battery-powered driver, we confirmed that the instrument was moving out—we had overcome the recalcitrant bolt and the old instrument could be removed. We were then able to install the WFC-3. The PGT was designed by the Hubble team and is now used on every NASA spacewalk on the International Space Station as a standard tool. In mission control David Leckrone, the NASA Hubble project scientist, remarked that the task “was tougher than we thought but it has ended just beautifully—although I just lost 5 years of my life, I think.” Indeed, the results from the WFC-3 are beautiful, as exemplified in figure 4, a photo of the stunning Eagle Nebula. FIGURE 4 The magnificent Eagle Nebula (M16), a star forming region in the constellation Serpens, as viewed by the Hubble Space Telescope wide field camera 3. The left panel shows the Nebula in the ultraviolet/visible wavelength range, and the right panel in infrared light, which reveals young solar systems embedded in the gas and dust of the nebula. Image credit: NASA/ESA. Why was the ground team so confident the bolt would come out? Engineers always ensure that the torque limiters used for servicing are calibrated before and after a mission. After the first repair mission the torque limiter used for the WFPC-2 installation was calibrated over a range of temperatures. At the cold end of the range it was found to be off calibration, providing as much as 53 ft-lbs of torque, very close to the fracture torque, instead of the design torque of 35 ft-lbs. It must have been cold when the astronauts installed the WFPC-2. One of the GSFC engineers in mission control knew about the calibration data and brought it to the JSC flight control team. The extensive collective engineering memory of the team, as captured in the Hubble knowledge management system, was critical for the installation of the WFC-3 and for all the Hubble repairs. Repairing the Space Telescope Imaging Spectrometer During the fourth spacewalk Mike Massimino had the task of repairing the Space Telescope Imaging Spectrograph (STIS). His first task was to remove one of the yellow astronaut handrails, which was blocking access to the panel over a failed power supply board. This was another tricky repair involving the removal of dozens of tiny screws. The handrail was attached by four #10 stainless steel socket head cap screws. Using the PGT with a 5/32” Allen drive bit, Massimino set about removing the screws. Unfortunately, he was accidentally triggering the PGT before inserting the bit into the head of the screw, stripping the hex features in the head. Three of the four screws came out, but the fourth was set solid and the handrail could not be removed. NASA plans spacewalks to a nominal time of 6½ hours, balancing the need to accomplish as much as possible with the duration of consumables in the suit and human endurance. For more than 4 hours this particular spacewalk was stuck in limbo. Without being able to remove the handrail the task could not be completed. Inside we studied the contingency procedures but knew this was not a case we had considered. While we waited on orbit, the ground team got busy. Engineers and technicians at GSFC accessed fittings, bolts, and the handrail from the mockups to figure out a way to remove the handrail, now held by one stripped bolt. They came up with a possible solution: fail the bolt in tension by pulling on the far end of the handrail. Through several pull tests they found that, with an average of 60 lbs of pull force at the top of the handrail, the bolt would fail. The operations team came up with a short procedure, ran it by the flight director, and called it up to us. “Do you think Mass[imino] can pull the handrail with 60 pounds of force to break it off?” “At this point I’m sure he could pull much more,” I said to my crewmates (if you know “big Mike” you know that he could have applied much, much more force). After applying Kapton adhesive tape to cover the bolt (which the ground experiment had shown could fly off), Massimino gave a mighty pull and, with a snap, off came the handrail. The rewards of enabling truly great and meaningful science are well worth the inherent risks of spaceflight. At over 4 hours into the space walk, Massimino and his spacewalking partner Mike Good finally proceeded with the STIS repair. In total they were outside space shuttle Atlantis for more than 8 hours. The new power supply board was installed and the instrument is still working and providing key science from Hubble. Conclusion The talent and detailed knowledge of the telescope, coupled with the ability to think quickly, organize a team to find a solution, and test that solution in real time allowed the Hubble team to succeed on every servicing mission. Thousands of people have had a hand in the success of the Hubble Space Telescope from conception to the on-orbit operations that continue today. I am one of the lucky few who have not only delved into the fascinating science that Hubble produces but also become an expert on its engineering and operations, working “up close and personal” on the telescope in space to upgrade and repair it during three space shuttle missions and eight spacewalks. Even with the inherent risks of spaceflight, the rewards of enabling truly great and meaningful science were, and still are, well worth the risk. My satisfaction comes from the knowledge of a job well done in the service of scientists, and the existential pleasure of engineering solutions to difficult challenges. Hubble may be the most productive scientific instrument to fly in space, and its legacy goes far beyond in the legions of engineers who had a hand in it and who have gone on to contribute to other programs in space exploration, national security, and many other fields. Who knows how long Hubble will last, or if the last space shuttle mission will be the last servicing mission (NRC 2005)? One thing is certain: Hubble will be remembered as a monumental scientific success, a marvel of engineering, and an exemplar of teamwork. References Gainor C. 2020. Not Yet Imagined: A Study of Hubble Space Telescope Operations. Washington: NASA. Giacconi R. 2008. Secrets of the Hoary Deep. Baltimore: Johns Hopkins University Press. Grunsfeld JM. 2014. Hubble: Mission impossible. In: Hubble’s Legacy, ed DeVorkin RD (pp. 61–72). Washington: Smithsonian Institution Scholarly Press. NRC [National Research Council]. 2005. Assessment of Options for Extending the Life of the Hubble Space Telescope. Washington: National Academies Press. Smith RW. 1989. The Space Telescope. New York: Cambridge University Press. Zimmerman R. 2008. The Universe in a Mirror. Princeton: Princeton University Press.  https://archive.stsci.edu/hst/bibliography/pubstat.html  https://www.stsci.edu/  https://www.invent.org/inductees/frank-j-cepollina  Hubble Tool Time series, https://svs.gsfc.nasa.gov/13174  The training tool and mockup now reside in the Smithsonian National Air and Space Museum (https://airandspace.si.edu/collection-objects/trainer- power-control-unit-hubble-space-telescope/nasm_A20120158000) . About the Author:John Grunsfeld is president/CEO of Endless Frontier Associates.