Conference Web Development

My notes on DevOps Kungfu

Chef Style DevOps Kungfu – Adam Jacob Keynote – ChefConf 2015

Safety: The ability for individuals to act without fear of unintended consequences

Contentment: Being satisfied with what you have

Freedom: The power or right to act, speak, or think as one wants without hinderance or restraint

Empowered Teams

  • Empowering ourselves and others to act, speak, and think as they need to with less hinderance.
  • Permission to act
  • Paired with the context to make good decisions
  • With leaders who care about your purpose
  • And share your beliefs
  • Do the right hard thing

Build consensus on important decisions

  • User your bonds to circulate plans with stakeholders
  • Incorporate their criticism and feedback into the plan
  • Present to the group
  • Ask for consensus

Sting value propositions

  • Pain killers, not vitamins
  • Make something people love
  • Needs not wants (one customer wants a feature; many customers need a feature)

Build a roadmap

  • Start with your vision
  • Align with customer feedback
  • Balance innovation with customer needs
  • Group them into themes, with an outcome for each
  • Distill those into features, and validate with customers
    • Your themes should hold
    • Your outcomes might hold
    • The features will not hold

Managing Risk

  • Small batches, near term hypothesis
  • Validation comes from customers
  • Introduce near-term volatility to gain decreased long-term risk

Validate theory arguments with work, try it and look at the outcomes


  • Weekly
  • Anyone with anything to show
  • Invite everyone
  • Record it and post it internally

Choose languages and tools that felt the job

  • We are all polyglots
  • Learning new languages and tools is one of the great joys of our industry
  • Small batch + iterative development protects you from rest

Continuous Integrations

  • Always integrate branches to master
  • They should be short lived, iterative branches
  • Fix the build when it goes red

The Four Eyed Rule

  • Never deploy anything unless someone else sees it first

Write Tests

  • Unit test (a single function)
  • Integration tests (multiple classes/units)
  • Functional tests (user-oriented, high-level, full-stack)
  • Smoke tests (quickly determine if the system is “working”)

Continuous Delivery

  • You can ship to your customer whatever product you want, whenever you feel like it, without drama.

One path per change

  • The way change moves through your organization is fixed

Code goes through the same path

  • Applications are code
  • Infrastructure is code

Focus on availability

  • Availability = uptime/uptime+downtime
  • Focus your efforts on reducing mean time to diagnose and mean time to repair
  • Failure is inevitable, it is how you respond and react to it that matters most

Collect Metrics

Plan for Capacity

  • Identify key metics
  • Put them on a graph
  • Set a limit
  • Plot a trend line
  • Expand your time horizon

Only Alert on What is Actionable

  • Get the attention of the right humans
  • As few alerts as possible
  • Rerouted to the people who can take action
  • Start with one alert “Is the system up?”

Practice Incident Response

  • Observe, Orient, Decide, Act
  • Orient is the step that matters most
  • The First Responder is the default Incident Commander
    • Decides who to do next
    • Coordinates resources
    • Communicates status
    • Not about rank
    • There is only one Incident Commander

Incident Lead to Post Mortems

  • Invade the space: we are to learn, not to blame
  • Describe the incident
  • Establish a timeline
  • Identify contributing factors
  • Describe customer impact
  • Describe remediation tasks
  • Describe improvement tasks for response process

Use Scalable Systems Design

  • Autonomous actors, responsible for themselves
  • Making progress toward their goal
  • Making clear promises to other actors
  • Having those actors evaluate the quality of those promises

Have Humility:

  • About your code
  • About your position
  • About your knowledge

Design Code to Favor

  • Simplicity
  • Extensibility
  • Re-use

Safety in Technique

  • Remember your principles
  • Practice your forms
  • Use your discernment

How to start DevOps

Stage 1 – Problem Selection

  • Small enough to have a meaningful iteration in 8 weeks
  • Vertical – not Horizontal

Stage 2 – Purpose, Beliefs, and Teams

  • Write down your purpose
  • Write down your beliefs
  • Empower the team to act
  • Be available for the context that the team will need (because we didn’t know everything when we started)

Stage 3 – Product Development

  • Write down your value proposition
  • Build a roadmap (Themes, Outcomes, Features)
  • Include delights
  • Focus on simplicity, extensibility, and reuse
  • Take short iterations of the problem

Stage 4 – Iterate on Features

  • Iterate on features
  • Manage risk through small batches, validate with customers
  • Choose languages and tools that fit the job
  • Ignore scale
  • When arguing about theory, focus on execution
  • Demo every single week
  • Use source control
  • Have a bug database
  • One workflow for change
  • Four eyes
  • Continuous integration
  • Continuous delivery
  • One test at a time (Unit, Integration, Functional, Smoke)
  • Use scalable systems design
  • Operate
    • Focus on availability
    • Collect metrics
    • Plan for capacity
    • Alert on what is actionable
    • Run incident response
    • Hold post mortems

Stage 5 – Deliver

  • Final Demo
  • Hold Retrospective