wickett

#DevOpsDays Notes from Nick Galbreath’s (@ngalbreath) talk on DevOpsSec at DevOps Days Austin.

• Kicks off with “Trust but Verify” which is a saying I use regularly.  
• Nick, also scopes the talk to AppSec for the audience.
• Ops and Security have commonalities, both have latent problems: Ops has failure that has yet to happen, Security has unexploited vulnerabilities. 
• Also, both Ops and Security have a “say no” reputation
• Also, MTTR is for security as well as ops.
• Blending the cultures of DevOps and Security, you have to think about how fast can you deploy and build your security stuff: firewall, VPN, DB, schema changes, apps, app patches…  Deploy cant be limited to just your app, it has to be everything in the environment.
• Nick also brought up deployinator and its impact at Etsy.  [personal note that I need to do a full out post on this]  The usage of deployinator and the culture that brings with it, leads Nick to the following:
• “Being able to deploy quickly is my #1 security feature”
• Nick got approval to hire some extra firemen (aka security guys) to sit around waiting for fires… His response was that he is more worried about the house burning down without knowing.  How do we catch the fires before they get huge?  Firemen are useless when the house is gone.
• Security will care about events that Ops won’t notice and Devs wont test.  SQL injection is a perfect example because a couple of SQL queries will dump your schema, db, … all of it.  Ops alerts wont fire and likely your dev team didnt test for it.
• This leads Nick to talk about Attack Driven Testing (Login errors, Server errors, core dumps, CSRF Failures, XSS, Password failures)
• We should use assert to positively verify and test our system/code. (Ports, processes, users, …)  [My Note: I think tying with BDD (cucumber) would be sweet for this and I am doing a talk on Rugged DevOps shortly about this.]
• Another rule at Etsy is to not give your customers a virus.  They use their continuous integration environment to run AV on the code using ClamAV.

Michael Cote (@cote) goes through the history of crowbar and how they got it done at Dell.

• There are two types of people in the world… those that understand DevOps and those who dont.  They had to attack the competing ideas against crowbar internally and externally and did so with metaphors (soup vs. sandwich).  He also mentioned speaking through your customers and use their quotes—let them explain what you are doing.
• Always be Coding, not educating
• Get customers and users ASAP
• Work the Iron Triangle
• Find the right content
• Hiding Out, things are easier when no one knows they should care.  Best way to do this is to only talk limited scope about what you are working on.
• Get by with just enough architecting and abstracting - Lean baby, lean. See Lean Startup book.
• Don’t open source a box of junk.  (My note: This is what I did with Gauntlet and yeah, this is probably right.  Release running code.)
• Market the right stuff

Great talk at DevOps Days Austin by James Turnbull.  Here are a couple notes that stood out to me.

• Teaching the auditors to read the puppet manifests gives them the power to figure out what is in the org.  Creating empowerment and common language between groups.
• Security Ops is not that different from regular Ops, they are just focused on one specialty.
• Some security people are some of the smartest people in the org and know how to code better javascript than your devs and they run ops better than your ops.  They have been hacking JS for a long time and they have been managing the checkpoint firewalls for years.
• Dev and Security needs to interface early, not at the end, which is often the case.
• James listed top three things he hated about being in security: not being effective, not being happy, and not being liked. 
• Bridge Ops and Security.  Put security into the Ops escalation process, invite them to post-mortems and hook them into your metrics and data.  They are good at parsing data and detecting anomalies. 
• I wish I could have counted the number of times James said, “Security people are good at _______.”  He has a healthy, positive view of the security folks in the org.

Great talk by @kartar and really hits home.