allardjd 1,853 Posted June 11, 2015 Report Share Posted June 11, 2015 This report brings back memories of something similar in my former life. Some of you may know my career was in commercial nuclear power, not aviation. To set the stage, during my time at the power plant, my company embarked on a program to upgrade a large number of aging electronic and electro-mechanical controllers to more modern, CPU-based replacements. It was done because the vendors who originally supplied them had moved on to newer, more capable devices and identical replacement controllers, parts and vendor support were becoming sparse. It was also anticipated that the newer model controllers would provide better performance and additional functionality that was beyond the capability of the current hardware. Almost all of the several hundred controllers that would be replaced were in nuclear safety related functions. Naturally any such design changes to a licensed nuclear facility require the scrutiny and approval of the US Nuclear Reguatory Commission. I recall two particular issues. One was called "common-mode failure". This was strictly a design issue, not a maintenance issue. Almost every controller performing a safety related function had at least one other in parallel, performing the same function. Common-mode failure referred to the possibility of an unforeseen combination of inputs (from sensors and/or human operators) which the software that controlled the device was not able to handle, causing the device to malfunction or lock-up. The concern, of course, was that the parallel devices that were there to provide redundancy were running the same software. Such a software error would similarly affect any number of redundant controllers, causing all to be unable to perform the required function, defeating the required redundancy. This was entirely a software design and testing issue but required a heavy-duty test and certification program. The other concern was simply software quality control (actually firmware, in these devices), i.e. assuring that the correct software was correctly installed in the devices at all times. Revisions to the firmware were fairly common, including adjustment of certain non-operator adjustable technical parameters in the device - think of "gain", for instance, or alarm setpoints. Those "tuning" kinds of things required re-setting and adjusting on a fairly frequent basis. In order to deal with that, a very rigorous in-plant software control program had to be developed and implemented. That program included a formalized training program and certification of the trained individuals in the process. Only such certified technicians were ever allowed to perform any kind of maintenance on the devices. Additionally, a strong, formalized revision control process was developed and implemented. That program was the source of record for the current approved software version and all firmware setpoints for each installed device, taking into account the installed purpose of the device (identical controllers were used for many different functions, requiring different software and parameters for different functions). Typical nuclear quality control and quality assurance measures were built in throughout the process, including working to written, pre-approved procedures and independent verification of critical steps. Any changes in software version or setpoints was subject to prior review and approval by the Plant Nuclear Safety Review Committee. Ad-hoc, on-the-fly changes were not permitted, ever. If what is reported is accurate, some aviation equivalent of the Software Quality Control function I refer to appears to be responsible for the A400M crash in Spain. By all appearances they either didn't have one, had one that didn't cover all the bases, or had one but didn't follow it properly. Again, if this is really what's behind the crash, this is good news for Airbus. This kind of thing is infinitely easier and quicker to fix than design changes to the physical aircraft. Aviation and commercial nuclear power share this - if you're going to do it, you'd better do it right, first time, every time, all the time, or very bad things can happen. Driving out human error completely is impossible, but good management, good programs and a strong safety culture can keep it to acceptably low levels. John 1 Link to post Share on other sites
brett 2,314 Posted June 11, 2015 Report Share Posted June 11, 2015 My worry is that not only humans making mistakes with the computers but the computers themselves failing. When a part fails on my car it runs like crap but when the computer goes it doesn't run at all because it controls to many things. 1 Link to post Share on other sites
allardjd 1,853 Posted June 11, 2015 Report Share Posted June 11, 2015 In mission-critical functions, there's often some hardware redundancy built in. The concept of single-failure proof is widely used in safety-critical functions. That was at the root of the concerns about common-mode failures - the redundant stuff fails too, and then you are well and truly screwed. It's necessary that the common-mode failures be aggressively addressed and eliminated in the design phase of the software. John 1 Link to post Share on other sites
MartinW 0 Posted June 12, 2015 Report Share Posted June 12, 2015 My worry is that not only humans making mistakes with the computers but the computers themselves failing. When a part fails on my car it runs like crap but when the computer goes it doesn't run at all because it controls to many things. The M-MMS system, that handles stuff like communications management, cargo handling, fuel management, has two dual redundant computers. The A400M also has 4 independent flight control computers. Unlike your car, it wont fail if one of them misbehaves. All fly-by-wire aircraft are designed this way. The A380 for example, has 3 primary flight control computers, plus 3 secondary flight computers, with dissimilar hardware and software. There's no defence against an idiot that wipes critical files though. Clearly this is something Airbus must address by protecting those critical files from accidental deletion. http://militaryaircraft-airbusds.com/aircraft/a400m/a400mabout.aspx Link to post Share on other sites
Andrew Godden 943 Posted June 12, 2015 Report Share Posted June 12, 2015 The M-MMS system.....has two duel redundant computers. No wonder they are falling out of the skies, the computers are dueling and killing one another. 1 Link to post Share on other sites
allardjd 1,853 Posted June 12, 2015 Report Share Posted June 12, 2015 Even the now-ancient Space Shuttle had three. If I recall correctly they all ran a continuous polling routine and compared one another's outputs. If one of the three produced an output that was different from that of the other two, that one would be immediately "locked out" by the other two. I guess it worked. John Link to post Share on other sites
MartinW 0 Posted June 12, 2015 Report Share Posted June 12, 2015 The M-MMS system.....has two duel redundant computers. No wonder they are falling out of the skies, the computers are dueling and killing one another. My opinion of someone who picks up on minor typos is........ Link to post Share on other sites
allardjd 1,853 Posted June 12, 2015 Report Share Posted June 12, 2015 A simple typo is one thing but sometimes they impart another unintended meaning that is kind of humorous, as in this case. The inadvertent inference of dueling computers is pretty rich. John Link to post Share on other sites
MartinW 0 Posted June 12, 2015 Report Share Posted June 12, 2015 I haven't said what my opinion is yet. It reminds me of "sweat" and sour sauce. Link to post Share on other sites
allardjd 1,853 Posted June 12, 2015 Report Share Posted June 12, 2015 Maybe we ought to get back on-topic here. Multi-computer redundancy and failure-tolerant control schemes are certainly germane to where this accident investigation has taken us. Can we continue from there? John Link to post Share on other sites
Captain Coffee 2,030 Posted June 12, 2015 Report Share Posted June 12, 2015 And I was trying to format a comment/joke regarding dueling computers and Blunderbusses Link to post Share on other sites
allardjd 1,853 Posted June 13, 2015 Report Share Posted June 13, 2015 Blunderbusses You're not casting aspersions on the Airbus line, are you? John Link to post Share on other sites
MartinW 0 Posted June 13, 2015 Report Share Posted June 13, 2015 Maybe we ought to get back on-topic here. Multi-computer redundancy and failure-tolerant control schemes are certainly germane to where this accident investigation has taken us. Can we continue from there? John A very good idea. Seems we haven't in the last couple of posts. so... So we know that the installation of software also deleted the vital torque calibration parameters for the engines. Each engine has an associated ECU of course. If an ECU detects the absence of the torque calibration parameters it automatically shuts the engine down, or at least reduces the engine RPM to idle. The pilots wouldn't receive a warning until 400 feet. This auto shutdown is to prevent issues with faulty engines unexpectedly powering up. What Airbus didn't envisage of course is this happening to 3 out of 4 engines. What I'm unsure of, is whether the guy that installed the software was culpable, for example by not following proper procedures, or if it was actually the software itself that he was installing that was bugged. Or perhaps the torque parameter config file itself was in some way bugged and thus vulnerable. Very disturbingly, it seems Airbus where aware of the fact that there would be no warning of deleted or faulty config files while on the ground. According to some sources, this was a safety issue that was raised last year. The claim being that the regulators approved it on the basis that the chances of failure were small. Quite clearly, Airbus need to modify the systems so that the ECU's warn them on the ground. Edit: It seems it was actually the computer system that was used to install the flight control software and update calibration data that was at fault. Airbus has instructed its European NATO buyers of the A400M not to use that software. European NATO buyers have now been instructed not to use the Airbus computer system that was used to conduct the software installation on the A400M, people familiar with the order said. Investigators are poring over maintenance records to see how safety procedures failed to notice the erased parameters. Without the vital data parameters, information from the engines is effectively meaningless to the computers controlling them. The automatic response is to hunker down and prevent what would usually be a single engine problem causing more damage.This is what the computers apparently did on the doomed flight, just as they were designed to do. "Nobody imagined a problem like this could happen to three engines," a person familiar with the 12-year-old project said. http://www.reuters.com/article/2015/06/09/us-airbus-a400m-idUSKBN0OP2AS20150609 Link to post Share on other sites
allardjd 1,853 Posted June 14, 2015 Report Share Posted June 14, 2015 I believe I'd start by adding a "take inventory" function to the software in the power-up process for the ECUs or FADEC controllers or whatever they use, and generate a caution light if anything important is not there. That aside, there's a hole in their software configuration control program. It may be a program problem or a human performance problem or a hardware/software problem, but a robust configuration management program should make what happened very nearly impossible. It's not hard, but it's got to be rigorously applied. I'm pretty sure they must realize all this by now. This is separate from software design and testing - that's equally important - but once the software is designed, tested and approved for production use, a program is needed to assure the right software version is correctly and completely installed, every time. John Link to post Share on other sites
allardjd 1,853 Posted June 14, 2015 Report Share Posted June 14, 2015 Well, someone is certainly optimistic....Airbus says US to be biggest customer for A400M military planehttp://news.yahoo.com/airbus-says-us-biggest-customer-a400m-military-plane-103700613.html ' "By the next decade at the latest, the US armed forces will be the biggest customer for the aircraft," Airbus chief executive Tom Enders told the weekly magazine WirtschaftsWoche in an interview...' I wonder if his calculation is before or after Germany immediately sells 13 of their now-mandatory purchase of 53, as they say they will do. Germany is currently the biggest customer. Remembering the famous/infamous KC-135 replacement story of a year or two ago, I'd bet against Mr. Ender's prediction. John Link to post Share on other sites
allardjd 1,853 Posted March 4, 2017 Report Share Posted March 4, 2017 I'm dredging up an old topic here to add a recent news article about the A400M. It seems Airbus would like to be relieved of their contractual obligations for delay penalties related to the delivery schedule for the A400Ms to their respective customers, to the tune of about a billion Euros this year. I believe the penalties are, in part at least, intended to compensate the customer countries if delays in the delivery of their aircraft require costly life-extension of older types or the lease or purchase of interim airlift capacity until they finally get the A400ms they contracted for. For the life of me, I can't see the logic that says that errors, oversights, technical deficiencies and wildly optimistic production schedule promises by the company are anyone else's responsibility other than the company's. Airbus CEO Tom Enders refers to the A400M program as a "financial sword of Damocles" over Airbus' head. While that's probably accurate, I struggle with the rationale for being allowed to evade the contractual requirements that Airbus signed on to with their customers in the last bail-out/contract update. A few years ago they got a boatload of new money, higher per-plane prices and schedule relaxation from the customers because of program problems up to that time. It seems like the same song, next verse, is being sung by Mr. Enders. If this isn't the worst aircraft development/production program ever, it's certainly in the top tier. Airbus is a great company and is very successful and very competitive in the civil aircraft market but they certainly don't seem to have brought much of that expertise, experience and good judgment across to this program. Military aircraft are different, and contracts with governments, vice airline companies are different. That's a lesson Airbus seems to be learning the hard way. http://www.dw.com/en/enders-warns-a400m-is-damocles-sword-for-airbus/a-37795514 Link to post Share on other sites
brett 2,314 Posted March 5, 2017 Report Share Posted March 5, 2017 How all that stuff works is foreign to me but I did get a kick out of his quote, " A400M customers could not force Airbus to "indefinitely carry" the financial risk and burden"(even if it's in the contract?). I can see the lawyers on all sides rubbing their hands together greedily with big smiles on their faces. Aren't their any comparable aircraft out there that these countries can buy if Airbus defaults on the contract obligations? Link to post Share on other sites
allardjd 1,853 Posted March 5, 2017 Report Share Posted March 5, 2017 The closest competitors are the C-130J, which is smaller, and the C-17 which is larger. There's also a Russian hauler that's pretty close to the A400 size but probably isn't a favored option for western military purchasers. UK leased, then bought some C-17s to take up the slack and reportedly like them. I think maybe others have too but can't remember the details at the moment. The C-130J is still pretty popular and, like the C-17, is a known aircraft with a solid history - you pretty much know what you're going to get with one of those. For those two it seems to be a Papa Bear, Mama Bear story, one too much, the other too little, and for one reason or another these seven countries decided at one point that the size of the A400 is "just right" for their anticipated needs/budget. One problem is, due to weight bloat, it cannot carry what was originally anticipated/specified. Some of the military features too, have been left out and are scheduled to be back-fitted to the already built airframes and added to not-yet-built aircraft later. A few of the customer countries, most notably Germany, are pretty hot under the collar over that. John Link to post Share on other sites
allardjd 1,853 Posted February 1, 2018 Report Share Posted February 1, 2018 The next chapter... "NATO buyers to meet with Airbus over billions in A400M fines" "WASHINGTON — Airbus will meet with several NATO members on Feb. 5 [2018] in London to discuss reductions to fines imposed on the company due to delivery delays and failing to meet contract capability requirements for its A400M Atlas military transport aircraft program, Reuters reports." "Officials from Belgium, France, Germany Luxemborg, Spain, Turkey and the United Kingdom, as well as Europe’s procurement agency OCCAR, will meet with Airbus in attempt to hash out an agreement capping financial penalties. Airbus received a $4.3 billion bailout from the seven countries in 2010." Link to post Share on other sites
dodgy-alan 1,587 Posted February 1, 2018 Report Share Posted February 1, 2018 As impressive as the A400 is iI can see this whole debacle becoming another Short Belfast. ie Big overweight aircraft that cannot carry the freight it was supposed to do. As much as I like the aircraft I cannot see it as having a long career. I remember seeing a mock-up of it at Farnborough years ago and it took forever to actually appear as a real aircraft. It could be a great aircraft if Airbus got their fingers out of their arses and sorted it out, However I think it may well be eclipsed quite early on. Antonov offered the RAF the AN-70 and it was also considered for the FLA requirement for NATO, from what I gather it could well have been a good contender against the A400. As it is ,the AN-70 also ran into production problems and to date only 2 prototypes have been built. https://en.wikipedia.org/wiki/Antonov_An-70 Link to post Share on other sites
allardjd 1,853 Posted February 1, 2018 Report Share Posted February 1, 2018 I doubt most Western countries are going to be very interested in buying Antonovs for their military airlift capability. John 1 2 Link to post Share on other sites
Captain Coffee 2,030 Posted February 1, 2018 Report Share Posted February 1, 2018 I bet the US would be happy to sell you all some C5's or c17s. And they seem to be without too many issues. Link to post Share on other sites
brett 2,314 Posted February 1, 2018 Report Share Posted February 1, 2018 No biggie, it's only digital money anyway, LOL. Link to post Share on other sites
allardjd 1,853 Posted February 1, 2018 Report Share Posted February 1, 2018 27 minutes ago, brett said: No biggie, it's only digital money anyway, LOL. Yeah, and since the customers are all military, it's just taxpayer money anyway and easily replaced. No need to be frugal (or Froogle). John Link to post Share on other sites
allardjd 1,853 Posted February 3, 2018 Report Share Posted February 3, 2018 On 2/1/2018 at 15:41, Captain Coffee said: I bet the US would be happy to sell you all some C5's or c17s. And they seem to be without too many issues. Maybe not. I understand the C5 is a hangar queen with a lot of maintenance issues. The AF is pretty much forced to keep them in the inventory because they are the only thing we have that can airlift an M1 Main Battle Tank. They'd like to get rid of them but can't and the production line is long since closed, so no more available. The C-17 is a different story - it is pretty universally loved by everyone who's had much to do with it and it has a very good maintenance and operational record. I think the production line for that one shut down too, a couple of years ago, but support and spare parts are still very much available. I think the C-130J line is still in operation, though it's not in the same weight-carrying class as A400M or C-17. John Link to post Share on other sites
Recommended Posts
Please sign in to comment
You will be able to leave a comment after signing in
Sign In Now