It's Time to Move to Modern Observability Tools and Progressive Delivery: Insights from Dynatrace

Observability has been around for as long as software engineering has been around. But it has clearly changed over the years!

If we look back 15 years, the classic 3-tier application stack (web server, app server, database) with application runtimes such as Java and .NET resulted in a new area of observability tools called Application Performance Monitoring (APM). Those APM tools focused on monitoring rather static infrastructure and the dynamic behaviour of the application runtimes and business transactions. As changes to those applications only happened occasionally, detecting problems and incidents could be done using simple statistical baselining.

Fast forward to 2023: Infrastructure is no longer static. It runs on a multi-layer virtualized stack on VMs, Container Orchestration Platforms (such as Kubernetes or OpenShift) or even in Serverless PaaS (Platform as a Service) offerings, distributed in global data centres. Applications are no longer 3-tier but rather a mesh of services spanning from self-developed cloud services to any type of SaaS-hosted third-party services. 

Why It's Time for a New Approach to Observability

Changes in applications (releasing new features) and adaptations to the virtual infrastructure to change load or resource consumption behaviour happen continuously. Deployments are now decoupled from releases by applying progressive delivery techniques such as canary deployments or feature flagging. 

With many more involved services and the shift towards dynamically releasing new features, the chances of becoming vulnerable to security problems and attacks are also alarmingly rising!

This explosion of complexity and the need to provide resilient and secure services requires a new approach to observability. One that can cope with constantly changing distributed virtual and cloud-based infrastructure, continuously changing services and configurations and the increased need to observe and mitigate any potential security vulnerabilities.

Without adopting modern observability practices many organisations will not be able to:

·         Identify and fix problems impacting end-user experience resulting in lost business

·         Identify resource usage inefficiencies resulting in higher operational costs

·         Identify and mitigate security threats jeopardising your business’s integrity

·         Identify and rollback progressive delivery changes threatening your systems’ stability

Modern Observability Has to Shift Left and Appeal to Developers

Traditionally, observability was implemented in production, providing operation teams insights into an unknown system (black box), to improve and identify potential problem areas in case of arising issues.

With the increased complexity of modern cloud-native environments, it is essential that observability is no longer seen as an afterthought. Observability must be included as a software development requirement—hence the term “Shift-Left”. Shift-left means that developers (who know the critical components and code best) must define what level of observability they need to determine whether their software runs as expected. This can be done through observability or code leveraging standards like OpenTelemetry to emit traces, metrics or logs.

Shift-left also means that observability data must be collected and analyzed in every stage of your software delivery pipeline. Software Quality Gates must include checks on whether expected observability data is collected successfully and whether systems are behaving as expected. The relevant observability data must also automatically forward to the right people and tools to make automated decisions in case of any anomalies.

Observability in the Context of Progressive Delivery

Progressive delivery—the technique used to decouple deployments from releases—is a key component in keeping your systems reliable, resilient and secure. Feature flags have become a very popular aspect of this in the last few years. The use cases range from deploying new end-user features to supporting operation teams with auto-remediating tasks by dynamically changing the code behaviour without having to redeploy.

Using feature flags only works successfully when making the use and impact of those flags observable. Therefore, it's necessary to make feature flags observable by default. Modern feature flagging as well as modern observability solutions provide out-of-the-box insights into which feature flags are used by which end user, how they impact the users’ experience or whether a new feature has a negative impact on the underlying dynamic infrastructure.

For compliance reasons, observability tools also keep track of when features were enabled and disabled to provide auditable tracing in case a feature had a non-desired impact.

Feature flag tools and their backends become critical system components. They have to be as resilient and available as all other components. Therefore, modern observability must also be applied to the feature flagging solutions themselves to ensure they are always able to deliver the right feature flagging configuration to the business app that is using them. (This is where the Flagsmith-Dynatrace integration comes into play.)

Conclusion: Modern Observability Leads to More Secure Innovation

Organisations that implement modern observability make it part of their software delivery requirements and enable developers to think about observability right from the start are those organisations that end up delivering innovation faster, more reliable and more secure.

More Observability Reading

• I recently contributed to Modern Development Practices in Banking: A Playbook, where I write about shifting left to modern observability in banks
• For more about observability and feature flags, check out this video walking through reducing release risk with AI-driven feature flag analytics
• Dynatrace has been a leader in modern observability and security for the past decade. To learn more about how Dynatrace can help your organisation innovate faster go here: https://www.dynatrace.com/