DevSecOps or Defeat

Mitchell Rubinstein will be speaking about DevSecOps or Defeat at HammerCon 2022 on May 19, 2022 - be sure to get your tickets here to hear him and our other exciting speakers before tickets run out! 

If you’re not already doing DevSecOps, you are dead in the water. In the context of cyberwarfare, without DevSecOps, you are unable to maneuver and likely to fall prey to attack. It’s telling that most of the time, we don’t even use the word, “cyberwarfare”. We call it, “cybersecurity”; a distinction that seems to indicate that our actions are exclusively defensive. And yet to the adversary that is attempting to “impose their will upon us, through the organized application of military force,” [1]  it sure looks like cyberwarfare to me. We must be better prepared in order to survive.

I don’t want to distract you by getting into cyber law, but part of the challenge is that at the policy level, we are still wrestling with how cyber fits into grand strategy. The threats are hard to attribute and consist of both state and non-state actors. Now where have we seen this before?

I submit that in fact, cyberwarfare looks akin to another form of warfare that we’ve seen before - that of “privateers with a letter of marque.” Privateers were armed ships owned and officered by private individuals holding a government commission and authorized for use in war, especially in the capture of enemy merchant shipping [2]. 

So, we’ve seen a similar type of warfare before and it’s nothing new. What can we learn from historical precedence to deal with these threats in a way that adequately protects us? Let’s start by taking a look at how we deal with cyberwarfare threats now.

Most U.S. Government systems have just one annual review to keep their Authority to Operate (ATO). And when that one review occurs, any gaps in cybersecurity can be waived with a Plan of Action and Milestones, or “POA&M” for which the software developers will get another whole year to fix. This means if a zero day is published at the right time, a program may have up to two years to patch the cybersecurity flaw and the adversary may have two full years to pursue it. 

U.S. Government employees reading this will say differently. “No, no,” they will argue “we would never take a whole year to fix a critical flaw. That’s not how that works.” They may offer a rebuttal of the way things should be working. But here, I can only tell you what I’ve observed (repeatedly) through my own eyes. With a two-year cycle to address a gap in your cybersecurity defense, you are giving your adversary two years to outmaneuver you and you are dead in the water.

The result of this two-year turnaround has effectively been a near total infiltration of our completely vulnerable cyber-infrastructure. This is evidenced by a myriad of high profile hacks, some of the more well-known are highlighted in the clippings below. For instance, hostile cyber forces sitting on our email for (you guessed it) two years.

In fact, this inability to compete temporally is so bad that last year the U.S. Government gave the FBI permission to hack into lawfully-operated cyber systems without the consent of the owner, in order to fix vulnerabilities [9] [10].

It’s hard to exaggerate what an escalation of government power that is. As you may be well-aware from recent events, most Americans are inherently allergic to the government telling them to do anything, even if it will keep them from getting sick. Or protect their security. In this case, the U.S. government is treating a symptom, but still not curing the disease. They are attempting to fix the consequences of an insufficient cybersecurity approach, but not changing the approach itself.

In the bigger picture, government solutions tend to be some variation of the same thing: Throw money at it! In fact the Biden Administration last year asked for $10B dollars to fund government cybersecurity [11].  The question we are left with is whether more money will solve our problems. Historical precedent says probably not. Just last summer, the U.S. Senate Homeland Security and Governmental Affairs Committee released a cybersecurity report which found seven federal agencies “have not met the basic cybersecurity standards necessary” [12].  And by the way, this report follows its predecessor two years ago, which found eight federal agencies failed to update known system vulnerabilities leaving their information open to hackers [13].  

The end result is we remain vulnerable to smaller, more agile adversaries. But perhaps the past can provide an answer. Where have we seen this before? What can we learn from the history of smaller, technologically inferior militaries defeating goliaths? History is replete with examples. Scholars of warfare may recall Afghanistan in 1842, Vietnam in 1975, Afghanistan in 1989, Somalia in 1995, Split-Yugoslavia in 1999, Afghanistan in 2021, and of course Ukraine today. Or, in the words of one of my mentors, let’s extract what has actually worked in combat and repeat that.

Now this article isn’t trying to “prove” what works in combat. I believe John Boyd already did that in his seminal works, A Discourse on Winning and Losing, Destruction and Creation, and Patterns of Conflict. John Boyd’s theory of winning is based around a fundamental cycle of naturally organized actions. Whether a person or a team, they observe the environment, orient themselves, decide on a course of action, and then take action. They then observe how the environment has changed, and the cycle begins again.  This Observe, Orient, Decide, Act (OODA) loop is depicted in the diagram below. The OODA loop is the same whether our frame of reference is a wrestling match, air to air dogfighting, large armies adapting to each other’s armor and weapons, or indeed, cyberwarfare technologies. 

The key point is that in combat, we cannot predict a deceptive enemy. And the combatant with the ability to adapt to the unforeseen events of the battlefield faster will prevail. So the way inferior forces win is by outmaneuvering the larger, slower, military force. The essence of maneuver being to act - not to directly attack enemy forces, but to generate and exploit some kind of advantage.

It’s always important to remember maneuver is not just spatial. It can be psychological, technological, or temporal. But especially important is maneuvering in time -- we generate a faster operating tempo than the enemy to gain a temporal advantage. And gaining that temporal advantage is so important because it builds upon itself with each cycle into an overwhelming advantage [14]. Napoleon Bonaparte famously said repeatedly to his lieutenants, "Ask me for anything, anything but time." What Napoleon was saying to his subordinate was that there are always setbacks. Terrain is sometimes lost to the enemy, but lost terrain can be regained. Forces are lost in combat, but can be rebuilt and reconstituted. But lost time? Once it has passed, time is lost forever. You will never see it again, and no General, however great, can win it back [15]. 

This form of warfare, where one maneuvers to gain an advantage, is called maneuver warfare. “Rather than wearing down an enemy’s defenses, maneuver warfare attempts to bypass these defenses in order to penetrate the enemy system and tear it apart” [16].  The preceding sentence explaining the philosophy of maneuver warfare is from Marine Corps Doctrinal Publication One - Warfighting. A masterwork of military doctrine, it was published and adopted as the official warfighting doctrine of the U.S. Marine Corps in 1989.

Something else happened in 1989. A man named Tim Berners-Lee submitted a proposal at the European Council for Nuclear Research (in French Conseil Européen pour la Recherche Nucléaire CERN) concerning information management. This proposal would later become what we know as the World Wide Web [17] [18].

Which brings up another tenet of maneuver warfare. War is both timeless and ever-changing. While the basic nature of war is constant, the means and methods we use evolve continuously [19].  We can see this is as true today as it was in 1989. The competitor that is able to adapt faster to the unforeseen events of the cyberwar will be victorious.

Hannibal hacked his way through the Roman Border Gateway Protocol, er, that is, the Italian Alps. Rommel hacked his way around the French Intrusion Detection System, er, the Maginot Line. If we are talking about the Great Firewall of China, Genghis Khan laughed at it in Mongolian.  Today, threats are no longer on the edges of our collective empire. They present themselves in our own houses and in our own personal vehicles. Every device connected to our Internet of Things becomes an attack aperture. Geographic location matters even less than before. Temporal advantage matters even more. Relatively slower, relatively static defenses, like cybersecurity authorities with annual reviews, are not going to survive.

“I’m very concerned that our nation has lost the ability to go fast. And we have adversaries now, and we see proof 

in those adversaries that they’re going faster than we are.” - Gen John Hyten, Commanding General of the U.S. Strategic 

Command [20]

Which brings us to why DevSecOps is our best chance for survival today. DevSecOps is really an approach, characterized by the disposition of our cyber talent, for time-competitive cyber warfare. DevSecOps, like John Boyd’s OODA loop, intrinsically regards time, measured in velocity, as a relevant metric of success. 

Looking at this cycle from the U.S. Department of Defense (DoD) DevSecOps Reference Design, you need to pay attention to how much security is baked into the process. DevSecOps is not just the inner cycle without all of the security. In fact, just so we’re all clear, maybe we should talk about what else DevSecOps is not. In doing so, I hope to clear up some misconceptions.

I hear a lot of people calling their development pipelines Continuous Integration / Continuous Deployment (CI/CD) pipelines. Yet they have zero cybersecurity built in. The question we should always ask our engineers is: “to where are you deploying the software, and who enforces the cybersecurity?” If it’s just a development environment, or if it’s just staging, and not production, or (and heaven forbid) you are seeing a demo on localhost, then your engineers aren’t delivering anywhere. You have a continuous integration pipeline only. Nothing is getting put into production. And nothing is secure

DevSecOps is also not the same thing as US Government (USG) Software as a Service (SaaS). While the two processes are often conflated as the same, SaaS involves building the software using traditional software development methods in the cloud. In the USG, it also requires the application to be certified by a government program called FedRAMP. It is limited to one classification at a time, mostly unclassified.

SaaS applications are usually developed entirely at their intended classification level, which for classified applications probably means your engineers are in windowless rooms for 8 hours a day, every day. This often results in middling talent and therefore middling results. If you try to recruit a software engineer at the top of their game from Facebook, Netflix, Amazon or Google, and tell them they are now restricted to a windowless room for their workday, you’re not going to keep that top talent for very long. Importantly, SaaS takes well over a year to develop and deploy applications to production because it uses the traditional Risk Management Framework (RMF) of cybersecurity. Making them the very opposite of time competitive.  

As discussed, it’s this time element that matters most, and it’s the hard part. Because time-competitive cybersecurity is an ever-adapting target. It’s not just a checklist that one can turn green. If it’s difficult for specialists, it’s even harder for our non-technical customers to understand whether or not their system is defensible. Until they get hacked, of course. Then it’s very easy for them to understand that their system was not secure, and why a time-competitive posture is critical.

If you are the owner of a critical system, you should understand this: where the traditional Risk Management Framework used by the USG accredits the code once a year (this is called an ATO), the DevSecOps methodology accredits every single line of code as it is pushed. From “Hello world!” through every last merge request. This is (formally) called a continuous ATO (c-ATO) which uses automated testing and security. It exceeds the RMF and does it with every update. If we look at the platform like a factory, we don’t accredit the software itself. We accredit the factory and the gates within. And then, of course, teams are authorized to use the factory.

This is a picture from the Audi factory in Brussels, Belgium. Making an analogy, we’d accredit the robots and the programs that drive the robots. In this way, we know that what comes out of the factory is accredited every single time. If there’s any deficiencies, inspecting robots or gates would not let the car through. Similarly, a DevSecOps software factory consists of smart automation using tools that enable processes for rapid deployment and scalability. Cybersecurity functions and policy are enforced from inception through operations. 

By approaching mission applications this way, the DoD DevSecOps Reference Design brings a holistic cybersecurity stack, gaining complete visibility of all assets, software security states, and infrastructure as code. This is what we mean by “cybersecurity being baked in”. The expectation is that every single line of code passes. No waivers, or it doesn’t get deployed.

The teams that are authorized to use the factory are made of DevSecOps Engineers. And it should be acknowledged, finding DevSecOps talent is not easy. What used to be done with three people, a Software Developer, a Cybersecurity Engineer, and a DevOps Engineer, is now one person. The coordination delays and communication errors are eliminated. It’s all in one head using very powerful automation. Of course, they still have to understand that breadth of knowledge and as such, these are senior engineers. But think about having a Battalion operating at the cognitive speed of two echelons below. Not just at the speed of a Company or Platoon, but a Battalion operating at the cognitive speed of a squad.

In short, what DevSecOps enables is the Silicon Valley approach to software development. You probably don’t know what version of Google search you are using. Any modern Silicon Valley-esque cloud-native application is updated many times a day. This ability to move fast and break things with cybersecurity built in from the first line of code opens up a world of possibilities when it comes to mission critical applications.

Using DevSecOps, Mission Applications can be managed as a service, deployed within secure cloud environments (including classified clouds) on Kubernetes, and it can be accredited at multiple classification levels at the same time. DevSecOps enables rapid prototyping with delivery in days, instead of a year or more.  This comes with a tremendous savings in cost and a tremendous savings in time [21]. 

It also enables responding to feedback in relevant timelines. In one real-world case, I was with a small team of engineers observing a mission briefing at a U.S. Air Force exercise. One of the operators using our mission application turned to us and said, “You know what would be great? If the tanker plan tool had a [widget] that did [this thing].” The engineers were able to implement that widget and push it into production before the end of that same mission briefing. That’s the speed of relevance.

Of course, when the operators see such responsiveness, then the floodgates open. They start to feel invested in the application. Their ideas are actually being listened to and implemented. They can get the application they want and will actually use. Not some application that doesn’t get delivered until 3 years later, when they are on their next assignment, that nobody asked for, and is already obsolete anyway.

DevSecOps enables bug and security fixes in minutes instead of months. For example, twice during Exercise Northern Edge ‘21, the on-site Alaska team I was with was notified of a new zero-day exploit. Which meant they would be required to include a patch for the zero-day in their next code update. If not, the code wouldn’t pass the pipeline gates of the DevSecOps platform and therefore wouldn’t be allowed to be deployed. Our teams made the proper patches to address a previously unknown zero-day within 3 hours. This should be the standard.

One of most well-known DevSecOps applications I was involved with is a mission planning application for the U.S. Air Force. The USAF Web-enabled Information Dominant Warfare (WIDOW) Mission Planning application went from nothing, meaning 0 lines of code, to actually in use on unclassified networks by the USAF Weapons School, flying the largest exercises the Air Force flies, in 4.5 weeks leveraging a c-ATO. Someone claiming the ability to go from inception to operation in 5 weeks using traditional RMF would be laughable. WIDOW demonstrated this rapid development capability using the DoD DevSecOps Reference Design. WIDOW was designated the official mission planning cell application for the USAF by week 8. It was deployed to a Secret network by month 4. It was deployed to a Top Secret network by month 6.

This success has now been repeated many times within the DoD DevSecOps community. Minimum Viable Products (MVPs) are expected to be delivered in less than 3 months. Each cloud-native mission application is available to anyone in the U.S. Department of Defense, worldwide, with the URL and CAC authentication. It’s worth re-emphasizing that it’s not just the speed of delivery that’s important, it’s the cybersecurity. The cybersecurity is the difficult part. Audi’s are great products, the company probably uses a great agile process. The vehicles even have security built in. Of course, that level of security is totally inappropriate for warfare. If you’re not doing DevSecOps, you are entering the cyber battlefield in an Audi instead of an Abrams.

So how can the rest of the DoD and USG easily and quickly adopt this Silicon Valley DevSecOps approach?

The DoD has published the DoD DevSecOps Reference Design. This design is intended to provide DevSecOps while avoiding vendor lock in. It uses Free and Open Source (FOSS) Kubernetes with Cloud Native Computing Foundation (CNCF) and Open Container Initiative (OCI) Containers [22]. 

That’s a lot of acronyms so let’s look at this in a different way.

This eyechart is the Cloud Native Computing Foundation Landscape. Each of these tiles is a different tool or technology that performs a different function in the cloud. Some are very secure, some are not so secure. You could spend a long time trying to find a stack of these tiles to create a software factory. It’s hard to do it well and even harder to do it with an uncompromising cyberwarfare standard of cybersecurity.

A few short years ago, there were several innovation efforts. The U.S. consolidated these efforts into one in order to save time, money, and resources. The result is Platform One and the DoD DevSecOps reference architecture. In other words, the DoD has figured out a very good, very secure path through the Cloud Native Computing Landscape for you. Below are some of the technologies that are used in this stack.

Some highlights include:

●     Hardened and centrally accredited containers.

●     Zero Trust using sidecar container stacks. Behavior detection down to the function level.

●     Continuous monitoring. Continuous scanning. Continuous penetration testing.

●     Scalable microservices architecture using Istio service mesh.

●     And note that the architecture inherits 90% of the NIST 800 controls. Then uses hardened, centrally accredited containers on top of that.

Some of the services available to development teams include:

●     The DoD Centralized Artifact Repository affectionately known as the Iron Bank, which is the repository for hardened and centrally accredited containers. The basic building blocks of Secure Mission Applications.

●     Party Bus is a multi-tenant DevSecOps environment where your mission application lives in an environment with several other organizations. You all share the time-competitive cybersecurity services.

●     Big Bang is a single tenant DevSecOps environment, where your organization is the only one and the platform and the time-competitive cybersecurity teams work only for you.

Of course, adopting DevSecOps is not going to be as easy as simply reading the DoD DevSecOps Reference Design and having your team stand up a Big Bang.  As of this writing, there are four USAF software factories with accredited continuous ATOs. Obviously, Kubernetes is a relatively young technology and there just aren’t enough DevSecOps engineers to go around. Fortunately, Platform One initiated a reseller program to help meet the demand. There are now five officially authorized Platform One Resellers.

DevSecOps is bleeding edge tech. Well, bleeding edge for the U.S. Government anyway. Silicon Valley wouldn’t call this bleeding edge. After all, the Agile manifesto is now 20 years old! Meanwhile the USG is still having a hard time getting rid of Waterfall development. Or what some colloquially call Agilefall, where the development teams follow waterfall methodology, but use agile terminology. Platform One assesses less than 1% of the US DoD has adopted DevSecOps. 

Why is adopting DevSecOps critical to your survival in the time-competitive cyberwarfare environment of today? It’s simple; our adversaries are already mastering it.

References

[1] Marine Corps Doctrinal Publication 1 - Warfighting

[2] "privateer ." The Oxford Pocket Dictionary of Current English. . Encyclopedia.com. 28 Feb. 2022 <https://www.encyclopedia.com>.

[3] https://www.bbc.com/news/technology-49488614

[4] https://foreignpolicy.com/2021/01/04/flawed-cybersecurity-is-a-ticking-time-bomb-for-the-balkans/  

[5] https://www.politico.com/news/2021/03/30/russia-suspected-emails-478541

[6] https://www.cnn.com/2015/03/10/politics/state-department-hack-worst-ever

[7] https://www.businessinsider.com/solarwinds-hack-explained-government-agencies-cyber-security-2020-12 

[8] https://www.cnet.com/tech/services-and-software/biden-administration-blames-china-for-microsoft-server-hack/ 

[9]https://www.washingtonpost.com/politics/2021/04/14/cybersecurity-202-fbi-launched-an-operation-wipe-out-hacker-access-hundreds-us-servers/ 

[10] https://www.theverge.com/2021/4/13/22382821/fbi-doj-hafnium-remote-access-removal-hack 

[11] https://www.washingtonpost.com/politics/2021/06/01/cybersecurity-202-biden-administration-aims-big-cybersecurity-spending/ 

[12] https://thehill.com/policy/cybersecurity/566107-senate-report-finds-major-cybersecurity-shortcomings-among-federal 

[13] https://thehill.com/homenews/senate/450100-senate-investigation-finds-multiple-federal-agencies-left-sensitive-data/ 

[14] Marine Corps Doctrinal Publication 1 - Warfighting

[15] https://www.resilience.org/stories/2006-08-08/anything-time/

[16] Marine Corps Doctrinal Publication 1 - Warfighting

[17] https://www.w3.org/History/1989/proposal.html

[18] http://info.cern.ch/Proposal.html 

[19] Marine Corps Doctrinal Publication 1 - Warfighting

[20] https://www.thedrive.com/the-war-zone/14564/you-have-to-hear-what-keeps-the-head-of-u-s-strategic-command-up-at-night 

[21] Information on the DoD DevSecOps Reference Design paraphrased from sources at https://software.af.mil/dsop/documents/ and from Nic Challain’s presentations, including: DoD Enterprise DevSecOps Initiative and Platform One (https://youtu.be/gR0Mzo7WDss)

[22] https://software.af.mil/wp-content/uploads/2021/05/DoD-Enterprise-DevSecOps-Reference-Design-v2.0-CNCF-Kubernetes.pdf 

Magazine Home | About CYBER | Author Guide | Masthead | Archive Articles