After saying for weeks the spacecraft and systems performed “as designed” after recovering from a Mission Elapsed Timer issue just 31 minutes after launch and point-blank saying that no other anomalies occurred on the mission, Boeing and NASA have now admitted that two more serious issues — one of which could have led to the Loss Of Vehicle — occurred during Starliner’s first mission.
The public admission came only after the Aerospace Safety and Advisory Panel (ASAP) publicly outed the two companies on Thursday afternoon, sending NASA and Boeing scrambling to hold a press conference and release information for “transparency”.
In a statement on their website on Friday, 7 February, NASA revealed three serious anomalies on Starliner’s truncated two day Orbital Flight Test back in December.
According to NASA, “Following the anomaly that occurred during the December Boeing Starliner Orbital Flight Test, NASA and Boeing formed a joint investigation team tasked with examining the primary issues which occurred during that test. Those issues included three specific concerns revealed during flight:
- An error with the Mission Elapsed Timer, which incorrectly polled time from the Atlas V booster nearly 11 hours prior to launch.
- A software issue within the Service Module Disposal Sequence, which incorrectly translated the Service Module disposal sequence into the SM Integrated Propulsion Controller.
- An Intermittent Space-to-Ground forward link issue, which impeded the Flight Control team’s ability to command and control the vehicle.”
Of stark note, NASA directly stated that both software issues would have resulted in the “Loss Of Vehicle” had ground controllers not intervened.
While the circumstances surrounding the first issue with the Mission Elapsed Timer are documented, what came to light from ASAP on Thursday afternoon was a second major software coding error.
That issue’s existence was pointedly denied by Boeing’s Ramon Sanchez, Senior Operations Lead for Starliner, at a 15 January media event in the Commercial Crew and Cargo Processing Facility at the Kennedy Space Center where the Starliners are built.
During that event, Mr. Sanchez was asked if any anomalies other than the one with the Mission Elapsed Timer occurred. His answer was a pointed and definitive, “No.”
In a teleconference Friday afternoon, NASA’s Doug Loverro, Associate Administrator, NASA’s Human Exploration and Operations Mission Directorate, deflected, saying the second issue “wasn’t an anomaly because we found it and fixed it” — directly contradicting a NASA statement that repeatedly called the two software issues anomalies.
Mr. Loverro continued, saying Boeing and NASA did not have to disclose the second issue to the media or the U.S. taxpayers because “We fixed it. You wouldn’t want us talking about something that didn’t happen.”
The comments from Mr. Loverro came minutes after Jim Chilton of Boeing admitted they would never have found the second software issue had the first issue with the Mission Elapsed Timer not occurred.
It also followed comments from Mr. Loverro himself where he said the software errors were just symptoms and the real problem was the “numerous process escapes” in Boeing’s software design and testing.
According to NASA and Boeing in the Friday afternoon teleconference, the second software issue was caught just hours before it would have destroyed Starliner when it attempted to reenter the atmosphere.
Of note, Boeing’s Mr. Chilton seemed to disagree that Starliner would have been destroyed had the second software error not been found.
When pressed, Mr. Chilton of Boeing confirmed that the second software issue would have resulted in the Service Module colliding with the Starliner capsule after separation for landing.
He then added “nothing good would have come from it” while NASA Administrator Jim Bridenstine added “no one knows what would have happened” — directly contradicting his own agency’s written statement published on the NASA website earlier Friday.
Regardless, at the time the second software issue was found, Boeing teams, working with NASA, transmitted up a software patch to prevent the issue from happening.
According to NASA, “The team found the two critical software defects were not detected ahead of flight despite multiple safeguards. Ground intervention prevented loss of vehicle in both cases.
“Additionally, breakdowns in the test and verification phase failed to identify the defects preflight despite their detectability. While both errors could have led to risk of spacecraft loss, the actions of the NASA-Boeing team were able to correct the issues and return the Starliner spacecraft safely to Earth.”
NASA went on to chastise Boeing for not identifying the critical and safety concerning software issues despite “numerous instances where the Boeing software quality processes either should have or could have uncovered the defects.”
When asked in the Friday telecon what role NASA should have played in ferreting out these issues in agency reviews of Boeing’s software before flight, NASA representatives admitted that this is part of the investigation and that greater oversight and fixing the oversight errors present are needed.
Regardless, the two major software issues will now require 11 top-priority “systemic corrective actions” — with more to follow as the investigation continues.
“We do expect to have those results at the end of February, as was our initial plan. We want to make sure we have a comprehensive understanding of what happened so that we can fully explain the root causes and better assess future work that will be needed.
“Most critically, we want to assure that these necessary steps are completely understood prior to determining the plan for future flights.”
That last statement is the first walk-back from NASA of the agency’s passive assertion that NASA might allow Boeing to proceed with a Crew Flight Test next instead of re-flying the uncrewed Orbital Flight Test.
However, in the telecon Friday, both NASA and Boeing refused to comment on the need to refly the Orbital Flight Test, with Boeing even going so far as to assert that test flights aren’t performed to find errors or issues or validate fixes.
Mr. Chilton described the software issue as a “flat tire.” He said, “You don’t put a spare tire on your car to see if it’s flat. You do a pressure check. [The software fixes] is a spare tire check.”
NASA also stated that they were launching an Organizational Safety Assessment of Boeing’s work on the Commercial Crew Program.
“The comprehensive safety review will include individual employee interviews with a sampling from a cross section of personnel, including senior managers, mid-level management and supervision, and engineers and technicians at multiple sites.
“The review would be added to the company’s Commercial Crew Transportation Capability contract. NASA previously completed a more limited review of the company. The goal of the Organizational Safety Assessment will be to examine the workplace culture with the commercial crew provider ahead of a mission with astronauts.”
The post Boeing & NASA admit multiple anomalies on Starliner mission appeared first on NASASpaceFlight.com.