Leadership Talks - Anomaly Response

April 1, 2008 By: Don Jewell GPS World

Don Jewell, contributing editor for GPS World's military and government section, interviewed Col. Mark Crews (pictured above), chief engineer at the GPS Wing, U.S. Space Missile Command, regarding GPS anomalies that occurred in October 2007 that resulted in a great deal of concern in the international user and monitoring community.

Editor's Note: Present with Col. Mark Crews (MC) and Don Jewell (DJ) during the interview were Karl Kovach, (KK) and John Berg (JB), from the Aerospace Corporation and LaTonya Lofton-Collins (LC) from the GPS Wing Public Affairs office.

Many readers may not be aware, but between Oct. 8 and Oct. 10, 2007 there was what could have been classified as a significant anomaly in the GPS constellation, but for the quick action of an integrated team of government and civilian contractor team experts. This occurred almost three weeks after the extremely successful transition of the GPS constellation from a legacy mainframe, ground-based Operational Control Segment (OCS) to a new distributed system created under the GPS Wing’s Architecture Evolution Plan (AEP). OCS tracked all of the GPS satellites, collected information from their broadcasts, and computed satellite orbits, among other tasks; the AEP system accomplishes the same tasks and more only faster and with more interoperability. What is now being termed an upload anomaly caused a great deal of concern in the international user/monitoring community, especially among U.S. allies, and immediately the rumors started flying. To get the real story I spoke with a friend, an engineer and professional colleague, Col. Mark Crews, currently serving as the chief engineer at the GPS Wing at SMC on Los Angeles Force Base in southern California.

DJ:   Col. Mark Crews, first of all thanks for agreeing to this interview and also thanks to your staff that is here supporting you today; and a special thanks to Ms. LaTonya Lofton-Collins for helping us with the Public Affairs requirements. She has made these interviews an easy process numerous times and I want to thank her for that.

Now moving on, we want to be absolutely sure and get the facts down correctly. I’m not sure if we discussed this before, but who was it, in the government, USAF (U.S. Air Force) or DoD (U.S Department of Defense), that first realized there was a problem? Was it the 2SOPS, (2nd Space Operations Squadron at Schriever AFB in Colorado) or the GPS Wing at SMC? Just who discovered the problem or anomaly first?

MC:  Yes, 2SOPS observed the anomaly from the telemetry and space vehicle responses during the processing of the upload.

DJ:  Good, at least the operators noticed the problem first, but I guess what I really want to know is, does the system work as advertised? Was the GPSOC process working and did any of the affected users call the GPSOC, which is also located at Schriever AFB?

MC:  Yes, the GPS Operations Center (GPSOC) handled user reports as soon as they came in.

DJ:  And what was the first indication that there was a problem?

MC:  The first indication would have been that received by 2SOPS both in the (S-band and the L-band) in the course of the NAV (Navigation Message Update) upload, which was interrupted. That would have been the first indication, which would have been just about immediately.

DJ:  So the 2SOPS discovered the problem through the link to the ground station that was accomplishing the upload at the time?

MC:  Yes, based on the telemetry feedback.

DJ:  Now Col. Crews as I understand it, the satellites in question continued to broadcast. Was that something that 2SOPS could/should have controlled?

MC:  2SOPS, as soon as they discovered the problems in telemetry, began taking actions to resolve the anomaly. They took prompt troubleshooting and resolution steps to bring the space vehicle back into good health.

DJ:  And am I correct in assuming that out of the 31 active satellites currently on orbit it was just the Lockheed Martin built IIR satellites and the ITT payloads that were affected?

MC:  Yes, the anomalies affected five IIRs and one IIR-M. The GPS constellation is currently made up of 14 IIA satellites, 12 IIR satellites, and five IIR-Ms satellites.

DJ:  And am I correct in assuming that it really depended on the memory state of the satellite payload or what mode the satellite was in at the time an upload was begun, as to how much memory it had available. Is that a correct assumption?

MC:  The source of each anomaly was not the navigation payload memory size. It turned out there was a memory logic constraint that was exceeded and caused the anomaly.

DJ:  Since it was a memory logic constraint on the payload; is that how the ITT Space Systems Division became involved? If so when did you involve them; in what part of the troubleshooting process?

MC:  ITT became involved as core members of the troubleshooting team. ITT built the navigation payload for all the IIR GPS satellites. Two of the Wing’s technical experts from Aerospace, John Berg and Karl Kovach [who are here with us today] were also part of the troubleshooting team.

JB:  2SOPS has on-site support from both Lockheed Martin and ITT, and of course the IIR is a Lockheed Martin (LMCO) vehicle. The payload is built by ITT and both LMCO and ITT have on-site and factory engineers available. These experts were engaged immediately in the troubleshooting process. And they utilized recorded data from the new ground control system, Architecture Evolution Plan, to determine the exact problem. So 2SOPS had excellent support right from the onset.

MC:  ITT has an engineering model of the payload; they’re able to replay the uploads and determine, via [a] troubleshooting process — a very rigorous process — what the exact state of the satellite message was, the navigation messages, the PRN (pseudo-random noise) codes, whether they went to non-standard code or remained in standard code and so on. So ITT was able to conduct a highly-effective troubleshooting process in their lab. Although the anomaly occurred in approximately only one-of-every-four IIR uploads, ITT was able to troubleshoot it in pretty short order.

JB:  It’s the kind of problem where, if it occurs each and every time, it’s easier to isolate the cause. When problems occur intermittently, as in this case, determining the pattern and the root cause can be challenging. ITT did a great job in supporting 2SOPS on the anomaly resolution.

DJ:  That was a great explanation. Thanks. Now Mark, what users were affected? Are you able at this point to elaborate on that?

MC:  Well, yes, we can, and it’s hard to tell exactly who was affected as it depends on how the receivers processed the signals that were being transmitted. It’s clear that when the anomalies occurred, virtually no receiver would acquire a signal from the satellites that were undergoing the anomaly. So that’s without question and the reason was the navigation message didn’t pass any parity check since the message was a random sequence of ones and zeros. So, no user equipment could acquire the affected satellite as it came into view. In contrast, user equipment that was initially tracking the satellite and did not check for parity errors in the message, were “coasting” on the PRN code, ignoring NAV data. So it’s not clear exactly which set of user equipment may or may not have been affected. However, it’s important to note there are standard guidelines such as the RTCA [Radio Technical Commission of America] Minimum Operational Performance Standards (MOPS), which preclude the use of GPS signals when there’s an invalid NAV message. In order to reinforce this practice, the GPS Wing has taken actions to more clearly notify users in the Standard Positioning Service Performance Standard (SPS PS) update. This notification is already contained in the Precise Positioning Service Performance Standard (PPS PS). In addition, we’re taking action in our interface control document (ICD); we have a proposed interface revision notice for IS-GPS-200 — the signal-in-space ICD for the L1C/A signal. So, to answer your question, user equipment that processed the navigation message would have ceased using the GPS signal from that satellite during the anomalous time period. And so, it’s not exactly clear which user equipment may have tried to continue using the L1 C/A PRN code with an unambiguously-invalid NAV message.

DJ:  I think I understand that answer, acronyms and all, and it sounds to me like the various monitoring stations around the world would have kept monitoring the anomalous satellites, even if they detected an invalid NAV message?

JB:  The Air Force monitoring stations would normally continue to track the L1-L2 P(Y) signals; however, during the anomaly, the monitor stations lost lock on the L1-L2 P(Y) signals and were unable to reacquire the L1 C/A signal from the anomalous satellite.

DJ:  What about monitoring stations other than the handful of official Air Force monitoring stations? There are so many more of those, numbering over a thousand worldwide, including other government agencies and our allies. Have you received any information about how those monitoring stations reacted to the anomaly?

KK:  After the events, there were reports from other systems in Asia that coasted on the L1 C/A PRN code despite the invalid NAV message.

DJ:  The reason I ask, Mark is because I immediately, as the Government & Military editor for GPS World, started getting e-mails from people around the globe telling me there was something major going on with the GPS constellation; that suddenly they weren’t receiving some satellites. I gathered from my e-mail that up to five or six satellites were affected. Many people evidently thought the US Government had turned them off, or turned (SA) Selective Availability back on, or started maneuvering them across space. Frankly, some of the e-mail traffic was a little confusing. I am not sure anyone knew what to think ... and they thought of all sorts of nefarious scenarios. Of course some people will always think the worst.

That’s the point where I really started taking an interest and asking questions in earnest, because I started getting a plethora of e-mails almost immediately after the event, although I did not know what the event was at the time. So I was wondering how the individuals sending the e-mails knew there was a problem? If it is true that the anomalies shouldn’t have affected users, then I assume it was the monitoring stations that continued tracking the effected satellites that caused the reactions and were the reasons for the calls and e-mails?

MC:  Yes, to the best of our understanding, the monitor stations were in that class of user equipment that may have ignored the invalid navigation message parity and continued to process the Coarse/Acquisition PRN code from the satellite. But I just want to emphasize that this is not, in any way, related to Selective Availability. Rather, this was an unfortunate anomaly that occurred and should not be misinterpreted. It was due to a latent memory condition that had been dormant within the IIR satellite and on some particular uploads it happened to produce this anomaly. Fortunately, we were able to quickly troubleshoot the problem, restore normal operations, and prevent further recurrence of the anomaly.

DJ:  I was reasonably sure that the senders of the e-mails were wrong in their assessment of the situation, which is why I started asking questions. And what you’ve just explained confirms my first suspicion, that since none of the numerous e-mails were from our war fighters, none of our U.S. military receivers were affected. Is that correct?

MC: That’s correct, our military receivers, which processed the P(Y) Codes, know when to ignore certain satellites that may be experiencing an anomalous condition. During the anomaly, the military receivers lost lock on the L1-L2 P(Y) signals and were unable to reacquire the L1 C/A signal. So, they would have also ignored or dropped the anomalous satellite signal.

DJ:  So, at one point then, was the GPS constellation size reduced from thirty to about twenty-four or twenty-five healthy satellites?

MC:  No, the largest number of satellites that concurrently experienced the anomaly was two at one time, and that was only on the first two anomalies. Subsequently, only one anomaly occurred at a time, so we at most dropped from a constellation size of thirty satellites to a constellation size of twenty-eight satellites.

DJ:  That’s good to know. Now what part did AEP play in this anomaly scenario? Was AEP part of the causation or was it part of the remedy or both?

MC:  On September 14, 2007, the GPS Architecture Evolution Plan (AEP) ground control system underwent a flawless transition. The anomalies that we’re talking about occurred between the eighth and tenth of October of 2007 — three weeks after the successful AEP transition. The uploads in question were such that it did evoke the anomalous memory condition on the IIR satellites, but it was in no way due to a fault of AEP. In fact, the AEP ground control segment was able to help play a constructive role in positively troubleshooting and helping resolve the anomaly at the earliest point in time.

DJ:  Then this was an anomaly that had lain dormant in the system and just by switching to AEP the problem was found and corrected … which to me sounds like a good thing. What I’m trying to say is that fortunately AEP unearthed the anomaly and a team of GPS experts fixed it, correct?

MC:  Absolutely, this had been a condition that had been within the IIR satellites, undocumented, that had not been encountered under the previous ground control segment. Unfortunately it hadn’t been discovered, on the one hand, but on the other hand, after discovery of the memory constraint, AEP was able to help quickly resolve the anomaly. Once the anomaly was understood the Control Segment software engineers were able to quickly adjust AEP so as to avoid the condition causing the anomaly until a longer term fix may come about. In any case, there will be no recurrence of this anomaly, thanks to the successful operations of AEP and constructive troubleshooting of the Lockheed Martin and ITT staffs.

DJ:  You just answered my next three or four questions because I had a whole series of them concerning a reoccurrence; do you think there are any other problems or anomalies lurking out there that AEP might uncover? I know that’s a difficult question, as it requires you to gaze into your crystal ball.

MC:  Well, it’s always hard to foresee a potential problem, but I can tell you that the dedicated operators of the Air Force at the 50th Space Wing, the Second Space Operations Squadron and the contractor staff of the Boeing and Lockheed Martin teams in delivering AEP, together with the government, have taken every precaution to ensure that the rollout of AEP was completely successful. We will continue to be diligent in finding any errors through rigorous testing before any user would experience a potential failure. In addition, we have what is called an integrity failure mode and effects analysis, IFMEA, which is a process by which we carefully document any problems to ensure that action is taken, not only on the existing system such that no IIR satellites will ever experience this anomaly again, but also on future systems, such as the Block IIF, which is still under development and the Block III satellites, which are currently in the early acquisition process.

DJ:  Col. Crews you mentioned to me, in our meetings last week at the ION-NTM (Institute of Navigation-National Technical Meeting in San Diego, California) that the AEP LADO, or Launch Anomaly and Disposal Operations, procedures had worked perfectly; there were no problems there. But I meant to ask you about the year-end “Delta V” positioning maneuver for all GPS satellites. Have you performed any of those under AEP to date? I think I’m using the correct terminology … it’s the repositioning maneuver initiated annually to place the satellite back in the proper orbital slot or position.

MC:  Yes, the satellite orbit adjustments called Delta V’s are not done only at the end of the year; they’re done as required based on the slight changes in the orbits required to keep satellites in their assigned constellation slots. So, station-keeping the satellite keeps it in its correct orbit. That’s done as required.

DJ:  Has the 2SOPS actually performed a Delta V under AEP control?

MC:  Yes, absolutely, we’ve done several of those without any problems.

DJ:  Col. Crews, Mark, that’s really the majority of the questions I had about the anomaly, but I do have some questions about you and your future, and what you are most proud of among your many accomplishments during your tenure as the GPS Chief Engineer at SMC. Can we proceed now or do you want to let the other folks go?

MC:  Oh, we can go ahead and do this now, they can stick around.

LC:  We want to be nosy, Don!

[LAUGHTER]

MC:  They’re going to be nosy, man. If I told them to go, they’d act like they snuck out, but they’d still be here, Don.

[LAUGHTER]

DJ:  Okay, they’re going to stick around regardless?

MC:  Absolutely.

DJ:  Mark, exactly how long have you been the GPS Chief Engineer?

MC:  I’ve been the chief engineer exactly since July 5th of 2004.

DJ:  And I hear through the grapevine that you’ll be retiring soon?

MC:  Yes, I plan to retire as of September 1st of this year. [2008]

DJ:  Can you tell me, pardon me, tell us, about any plans you might have?

MC:  I plan to continue service within the private sector. It will probably be defense-related and more than likely in the space or global positioning systems business.

DJ:  What would you say is your proudest achievement as the Chief Engineer at the GPS Wing?

MC:  First of all, I’m proud to have worked with a world-class team which includes Aerospace, MITRE, the Systems Engineering and Integration team, civilian government personnel, and Air Force staff who are all incredibly dedicated to the Global Positioning Systems Wing. It has been both an honor and privilege to work with all of these patriotic and dedicated professionals. I’m extremely proud of the accomplishments we made with radio frequency compatibility. I had the privilege to co-chair numerous interoperability working groups to reach constructive agreements with Russia’s GLONASS, EU’s [European Union] Galileo, and Japan’s QZSS [Quasi-Zenith Satellite System]. These agreements include the finalization of the L1C signal design for the Block III satellites and that’s pretty remarkable. We decided on a signal design called Multiplex Binary Offset Carrier, which will be broadcast as the L1C signal via GPS III satellites as well as the E1 open-service signal of Galileo satellites. I look at this signal as a bridge in space, if you will, between nations. My contribution that will lead to the broadcast of this specific signal is one of my proudest achievements in the GPS Wing.

DJ:  Great Mark that’s all I’ve got for now and I sure do appreciate your time today as well as the time of your staff members who so diligently stood by and not once did they have to say, “What the Colonel meant to say was…”

MC:  Don, I really appreciate the opportunity for having this dialogue and thanks for the good work that you do, not only in GPS World, but also on the independent review team for GPS [GPS-IRT] and I look forward to continuing the dialogue and if you have any other questions, don’t hesitate to give us a call.