The Griswald Family Christmas Code Disaster
- Chris Wallace
- Sat Dec 21 2019
If Clark Griswald delivered software the same way he does Christmas Lights, it would probably go something like this.
With it being the season, it's time to watch the annual traditional Christmas movies. After watching the Christmas Light scene in National Lampoon's Christmas Vacation, I figured it would be a fun analogy for a disastrous software development lifecycle (and how to avoid it!)
The work is too big to accurately estimate...
...so too much is being done at once.
Lesson learned: Decompose a big problem into smaller pieces until they can be reasonably digested and implemented by anyone on the team.
The current solution is a mess of spaghetti code that needs to be untangled.
Lesson learned: Make room in your backlog to constantly refactor code into common patterns.
Not everyone is on board with the proposed approach.
Lesson learned: Even if it's an awful idea, a team needs to have each other's back and learn each other's mistakes as a unit. Rockstars or rebels within a team fragments the concept of shared code ownership, responsibility and accountability.
The dev tooling is unpredictable.
Lesson learned: If a solution has a poor test suite, lack of stable automation tools or a badly configured dev environment, don't expect clean and clear code to be written quickly. Invest time in constantly improving the tooling developers have at their disposal (and celebrate it).
The work is a mundane and repetitive grind...
Lesson learned: Keep code writing interesting, striving for as little repetition as possible. The more code is reused, the less lines available for bugs to creep in.
...which leads to occasional mistakes.
Lesson learned: Avoid opportunity for copy => paste => tweak errors. If a function is being copied and pasted with minor tweaks, the code has not been optimised and is widening its margin for error.
Dependencies are difficult to find and inject...
...so they're deferred until later.
Lesson learned: Common standards and practices are a must in development. The more common the pattern, the easier it is to identify the whereabouts of dependencies and how they should be used. Keep the amount of layers of abstraction under control in the application. The more layers an application has, the bigger the hit surface for bad practices and hacky code.
Stuff gets hacked and broken along the way.
Lesson learned: If something needs to be broken or hacked in order to satisfy a new condition, ensure automated test coverage is in place before refactoring and making changes. Where possible, write code that can be open to extension, but closed to internal modification (allow for behavior changes to be injected, rather than directly implemented internally).
Development deadlines are being driven by stakeholder demos.
Lesson learned: Own your own deadlines, and give stakeholders the opportunity to prioritize what they get delivered 'next'. Rather than getting doing a massive demo of 'everything' in X weeks, do intimate demos of 'everything so far plus Y' every week. The stakeholder gets a smaller feedback loop, and the delivery team works in controlled iterations.
There's unnecessary ceremony and theatrics around deliverables.
Lesson learned: Don't pop the cork on the champagne bottle until you see results. A feature is only complete when it's proven itself out in the field for the end user. Giving praise for a completed feature is a morale booster in the short term, but it's failures can demotivate people even more so. You can't end on a high-note when there's chances of a low-note!
And there's not enough (or any) testing before showing to stakeholders.
Lesson learned: Ensure you have a robust quality assurance and bug triage process in place. Something should only be demonstrated once it can be confidently released into production (no exceptions, caveats or random edge cases).
Stakeholders are unimpressed after their hopes have been raised.
Lesson learned: Transparency is key when delivering features, which is why having a shorter feedback loop helps. It allows the stakeholder to have increased visibility and understanding of current deliverable rather than expecting 'everything' to be done and perfect. 'everything' is only ever done when there's no more opportunity for innovation, which is never the case.
And there's been no time to triage bugs and defects.
Lesson learned: If something's not working as expected, take the time to sensibly document and analyse the defective behavior until the code works as expected. Not all bugs are equal. Some can be addressed post release, whereas others will be deal breakers.
Even the simplest parts don't work.
Lesson learned: Get the smallest piece working (and tested) first. Then increase the scope. A healthy development lifecycle is one that works in small successes.
"stupid questions" are dismissed.
Lesson learned: There are no stupid questions.
The solution is so complex, root cause analysis could be done anywhere.
Lesson learned: Keep the size of the solution under control, the more moving parts a solution has, the more time it will take to review and search for points of failure.
Uncontrolled side-effects make the code work...for some reason.
Lesson learned: When integrating with existing functionality, shared state or reusing code, really understand how that code or state behaves. If it's range of complexity is too broad, look at refactoring before trying to integrate with it.
The result is massively inefficient and not fit for LTS.
Lesson learned: Performance and load testing are a part of development and the earlier something is tested, the quicker it is to identify performance bottlenecks. Keep the code modular as possible so transactions, queries and processes can be tested and fixed in isolation.
The uncontrolled side-effects create a false sense of security.
Lesson learned: Before giving code your stamp of approval, it's important to break it first. Having tests in place to run code in isolation helps keep uncontrolled side-effects from interfering with the behavior, and highlights all the inputs and dependencies being used. Ensure you can make your code work as expected, but also ensure you make it fail as expected. Intentionally cause a breakage and assert the correct exceptions are being handled ("failing gracefully").
Stakeholders are brought back asap.
Lesson learned: Don't call 'an emergency demo' 2 minutes after getting a passing test. Review your code, have someone else review your code and try and break your code before stamping it as 'shippable'.
The uncontrolled side-effects show the code is unstable and unpredictable.
Lesson learned: When external factors outside your code changes are causing unusual behavior, it's time to start reading (and documenting ) what's been written so far to understand the instability and get it under control.
The code changes are stripped back massively until something works.
Lesson learned: This should have been done at the very beginning, get a small success first before trying to do 'everything' at once. Writing everything before running anything is not impressive, it's risky.
There's too many branching permutations to track, so "hoping for the best" becomes a thing.
Lesson learned: If you're crossing your fingers every time you run a build, it's a sign you need to reflect on your progress. Put the breaks on development, take a step back, look at all your changes so far, then decide how you can improve the code before adding more features to it. The cleaner and clearer your code is, the more confidence you'll have in it.
But all it takes is a second pair of eyes to share the technical vision.
Lesson learned: Your code is shared across your team. If you all have shared context and visibility, the faster it can be to resolve problems. If a single rockstar/ninja/rebel on your team has all the knowledge, it will take longer to identify points of failure.
It works as expected...
...and everyone loves it...
...apart from that one stakeholder.
Lesson learned: Never take it personally, you haven't built a masterpiece no matter how much effort you put into it. There's always room for innovation and improvement. If a feature is missing or not behaving as the stakeholder expected, address the situation and get it on the backlog. To avoid this situation, work in small increments so stakeholders get feedback sooner.
Time to repeat the lifecycle again!
Lesson learned: You've solved the problem, move on, get it shipped, have a positive retrospective on this long and drawn out story, and revel in the results post-release. Document the war stories you had along the way (write a journal if needed), then think how to prevent these problems next time. Constantly investing time to reflect on your progress in the short term helps measure your progress in the long term, it's a healthy motivator to see how far you've come and what you can still achieve.