Cloud Foundation: 3 valuable Players on the Field
Building Cloud Foundations can be cumbersome and tedious. In this post we’ll introduce 3 of our offerings in that field that will make your life easier.
If you are an Enterprise Architect and/or responsible for your organization’s Cloud Center of Excellence or Cloud Foundation Team this post is for you.
It will show you ways to accelerate your Cloud Journey, provide insights on Cloud Foundation best-practices and finally demonstrate how our offerings play together to help you build the best Cloud Foundation.
The Cloud Foundation Maturity Model – Getting Clarity
As a member of your Cloud Foundation Team, you may have come across the Cloud Foundation Maturity Model (CFMM). It’s a strategic model that helps you understand the capabilities you’ll need to build up during your Cloud Journey, regardless of which clouds you use, where you’re currently at or what approach to implementation you have taken.
You can consider it the coach on the playing field. It gives you guidance, shows where you’ll want to go and provides insights into different implementation options that will get you there.
We have used this model successfully with leading cloud-native organizations from various industries.
Building Cloud Foundations is an Iterative Process
Lots of organizations we work with run thousands of applications in the cloud, managed by a central Cloud Foundation Team. However, they didn’t get there within a day. Implementing a Cloud Foundation is best approached with an iterative approach. You start with the basics and implement more and more sophisticated capabilities over time as you get to know more about your users and their requirements.
The Cloud Foundation Maturity Model helps you to break the journey down into small steps that systematically build upon each other.
It’s not only about how far along the Cloud Journey you are, but also what type of vehicle you’re using
When Cloud Foundation Teams are faced with large-scale cloud migration projects, they know that whatever Cloud Foundation capabilities they build they’ll have to be highly automated in order to keep up with the upcoming scale of cloud demand.
That’s why we cover two dimensions when it comes to maturity within our Cloud Foundation Maturity Model:
- Journey Stage: Varying from Essential building blocks (1) to Industry Leading capabilities (5)
- Implementation Maturity: Varying from manual to fully automated
Assessing both brings you valuable insights when evaluating the current state of your Cloud Foundation and that’s exactly what the Cloud Foundation Maturity Model can be used for.
As a result, you’ll have a structured analysis of your Cloud Foundation, demonstrating your achievements as well as areas you’ll need to improve on a single page, which makes it a great tool for internal communication.
Collie – A Trustworthy Helper On Your Way To Cloud Foundation Maturity
Most organizations start their cloud journeys in high spirits on a green field. By the time you realize that you need proper cloud governance and decide to establish a Cloud Foundation, the once green grass has turned into a brownfield landscape. Collie is an open-source multi-cloud CLI that helps exploring and sorting, preparing for systetmatic management. It’s built for power-users, single player.
Collie CLI easily provides multi-cloud transparency on cloud tenants, corresponding access rights, tags and cost information. In regard to the Cloud Foundation Maturity Model it will help you to implement the following building blocks:
- Cloud Tenant Database
- Multi-Cloud Tenant Database
- Cloud Tenant Tagging
- Multi-Cloud Tagging Policy
- Monthly Cloud Tenant Billing Report
To fully automate your cloud foundation capabilities and establish collaboration between the Cloud Foundation and your DevOps teams, you’ll need to go one step further and use a platform like meshStack that reflects this organizational model.
meshStack – Your Cloud Foundation Platform
At meshcloud, we focus on enabling DevOps teams to use cloud and speeding up their applications’ time-to-market. Providing self-service access to native cloud tenants like AWS accounts, GCP projects or Azure subscriptions is critical to achieve this.
We know that every organization is unique and comes with specific requirements, especially in regard to the integration of existing IT Landscapes. Our aim with meshStack is to help you get the basics right and enable you to build upon them.
That’s why a lot of Cloud Foundation capabilities come out-of-the-box, while others can flexibly be adapted to your organizations’ needs.
In general there are 3 ways meshStack can help you to implement Cloud Foundation Building Blocks.
-
meshStack provides this Cloud Foundation Building Block out-of-the-box
Example: When using meshStack in a multi-cloud setup, you’ll have a Self-Service Multi-Cloud Tenant Database, out-of-the-box.
-
meshStack enables strong facilities for an easy implementation of that building block
Example: With Landing Zones and Services you’ll have everything you need to implement Modular Landing Zones.
-
There are established solution patterns that facilitate the implementation of this building block, when using meshStack
Example: There is an established solution pattern that will help you provide managed Service Accounts with meshStack.
In The End It’s About the People
Our offerings aim to help organizations throughout their entire Cloud Foundation journey:
- From understanding and defining the Cloud Foundation for your organization based on our Cloud Foundation Maturity Model
- to analyzing and understanding existing cloud landscapes with Collie CLI
- all the way to implementing large-scale Cloud Foundation Platforms to manage thousands of teams and applications based on meshStack.
However, establishing a Cloud Foundation within an organization is not only about building up the relevant capabilities and deciding on implementation options. It also is a huge organizational shift that requires involvement of all stakeholders and great communication across teams and up to the C-Level.
By allowing organizations to easily experience the power of self-service and providing means for internal communications, we want to enable Cloud Foundation Stakeholders to drive acceptance for change throughout the entire organization in order to pave the way for a new way to organize IT.
Cloud Tagging and Labeling on Azure, AWS and GCP - (Cheat Sheet 2022)
Are you looking for Azure tag requirements, AWS tagging documentation or you want to know how to use GCP labels? You have come to the right place!
In this post, we want to give you an overview of how different cloud platforms handle tags or labels.
You can see this as a cheat sheet to help you navigate the cloud tagging or labeling specifics of Azure, AWS, and GCP.
This post will go into detail on questions like:
- "How many characters can a tag have in Azure?",
- "How many tags can be assigned to one resource in GCP?" or
- "What characters are not supported for tags in AWS?".
So let's dive right in!
Using Tags in Azure
Here's what you need to know to get started with tagging in Azure:
- A resource does not inherit tags hierarchically from the respective resource group
- You can assign up to 50 tags to a single resource. If you need more, there is a little trick to add more: Creating tags with multiple values is a valid workaround.
- The maximum key length in Azure is 512. For values it's 256.
- Tags in Azure are not case-sensitive
- With tag keys you must not use the characters: < > % & / ?
How tags work with AWS
These are the AWS tagging specifications you need to follow:
- You can assign 50 tags per resource
- Tag keys must be unique for each resource and can only have one value
- The character limit for keys is 128, for values it's 256
- You can use the allowed characters across all AWS services: letters, numbers, and spaces representable in UTF-8, and the following characters: + - = . _ : / @
- EC2 allows for any character in its tags
- Tag keys and values are case-sensitive
- The
aws:prefix is reserved for AWS use. If a tag has a tag key with this prefix, then you can't edit or delete the tag's key or value. Tags with theaws:prefix do not count against your tags per resource limit.
Tagging with GCP
First: Google calls its tags in GCP "labels" - they are still tags though.
Let's see what requirements and restrictions GCP labels have:
- A resource can have up to 64 labels assigned to it
- Both keys and values have a maximum length of 63 characters
- Keys and values can contain: lowercase letters, numeric characters, underscores and hyphens
- You are able to use international characters
- Label keys must start with a lowercase letter
- Label keys cannot be empty
Cloud Tagging At a Glance
| Constraint/Platform | Azure | AWS | GCP |
|---|---|---|---|
| Max. # of tags | 50 | 50 | 64 |
| Max. tag name length | 512 | 128 | 63 |
| Max. tag value length | 256 | 256 | 63 |
| Case sensitive? | no | yes | yes |
| Allowed characters | < > % & / ? are not allowed | letters, numbers, spaces, and + - = . _ : / @ | lowercase, numbers, underscore, hyphens |
meshcloud offers a global management of tags in multi cloud architectures and makes sure they comply with the specific platform requirements.
To learn more about the meshcloud platform, please get in touch with our sales team or book a demo with one of our product experts. We're looking forward to get in touch with you.
Kubernetes Chargeback (splitting the bill made easy!)
Kubernetes (K8s) has become an extremely popular technology. With that Kubernetes chargeback and cost management have become pressing challenges: A recent survey by CNCF shows that there are 5.6 million Kubernetes developers today. A 67% increase compared to 12 months ago. 31% of all backend developers now use Kubernetes. This wide adoption of Kubernetes as an orchestration technology for containers makes challenges apparent - especially in larger organizations: Automated Kubernetes billing and cost management becomes more important.
What is Kubernetes exactly?
Kubernetes - aka K8s for short - is an orchestration technology for containers. You can use it for automating deployment, scaling, and management of containers. It works declerativly in a master and worker node setup. The master node checks the status and health of the worker nodes that run the actual containers. Everything is checked against a defined desired state and K8s makes sure to make this desired state come alive and stay alive.
K8s Challenges for Cloud Governance
Cost management for Kubernetes clusters is a pressing topic for many operations departments. Enabling cloud teams to work with the latest technology like Kubernetes is an important part of leveraging the cloud. But this has cloud governance implications: Cost management, security and compliance.
When running Kubernetes in a scenario in which you share Kubernetes clusters you run into billing issues. An example would be running Azure’s Kubernetes Service (AKS) as a multi-tenant cluster. There is no standard way of knowing how to split up the bill for the resource consumption of the Kubernetes cluster.
Onboarding internal customers and giving them a way to manage their Kubernetes namespaces, quotas and access rights is another cloud governance issue you might run into.
That’s quite a lot to tackle in terms of cloud management.
So here’s a short overview:
- Kubernetes cost management
- Security and compliance of shared K8s clusters
- Kubernetes namespace management
- Quota management
- Managing access rights
Splitting the K8s Bill with meshcloud
With the cloud foundation platform meshStack, meshcloud makes it easy to split the bill from shared K8s clusters. Many of our customers use cloud-native K8s services like Azure’s Kubernetes Service (AKS): They get one big bill from Azure every month and have not been able to fairly split the bill and charge their application teams based on actual usage. In order to do this, you can define a pricing catalog for AKS resource usage (like based on Persistent Volume Claims or the number of created Pods), that will be used to calculate a per month pricing for individual customers.
meshcloud makes accurate Kubernetes metering and billing possible and much more:
- Managing multiple projects,
- authorizing users on projects,
- charging the cost for projects to different teams, and
- setting usage quotas for projects.
Application developers want to (and should) develop applications
Application teams should do what they’re good at: Develop new applications and innovate. Administrative tasks often hold them back. DevOps teams want a managed K8s service where they don't have to set up the cluster themselves. They just want to develop applications. Typically we see OpenShift used for this type of scenario - OpenShift metering and management is also fully supported by meshcloud.
With meshcloud it is possible to lessen the administrative effort even more. An Openshift or AKS cluster can be shared easily across multiple customers for more effective resource usage. Project namespaces and access will be automatically set up so your teams are able to start deploying pods within minutes.
meshcloud is ISO 27001 certified
We at meshcloud are very pleased to announce that our organization is now ISO 27001 certified.
Our customers use meshcloud's cloud governance solution to manage thousands of their projects in the public and private cloud: Companies like Volkswagen, EnBW and Commerzbank trust meshcloud to provide a solid and secure platform for their cloud governance needs.
It's been very important to us that we can back up this trust with an ISO 27001 certification of our Information Security Management System.
What is ISO 27001?
The international ISO 27001 standard provides requirements and describes best practices for an Information Security Management System (ISMS). It is the only standard that sets out the specifications for an ISMS and an accredited certification demonstrates that an organization has identified its risks and has eliminated them or implemented appropriate countermeasures.
meshcloud's ISO 27001 certification
For us at meshcloud, it has always been of the highest importance to live up to the trust placed in us by our customers.
The ISO Certification was the next logical step for us as a company. We partnered up with ISO experts and aligned our operation to the high standard of ISO 27001:
- Developing internal ISMS competence
- Conducting risk assessment
- Implementing required controls
- Creating thorough ISMS documentation
- Training staff regularly
The audit of our ISMS included:
- Inspection of documents
- Observation and on-site inspection
- Interviews with staff members
ISO 27001 requires re-certification checks every year.
To learn more about the meshcloud platform, please get in touch with our sales team or book a demo with one of our product experts. We're looking forward to get in touch with you.
Launching Collie – Your Open-Source CLI for Cloud Governance
If you've been in touch with meshcloud, you'll know that we love sheep. And that's because clouds and sheep have a lot in common:
- They love to be free, rather than locked in a barn
- They love to be together, rather than wandering alone
- And they need a shepherd, to protect them from lurking dangers
With Collie, we're launching an open-source CLI tool that helps you as an enterprise architect or platform engineer to assume your shepherd responsibilities, giving you the control to steer your flock towards the next pasture.
More specifically, Collie enables you to:
- View all cloud tenants in one single overview - View your AWS Accounts, Azure Subscriptions, and Google Cloud Projects with their metadata with only one command.
- View billing information across all clouds - See what you spend per day, month, or week in all cloud platforms, including the right metadata. Includes support for CSV, YML and JSON.
- Analyze tag inconsistencies - See at a glance what tags are used, by which tenants (and which not), and what potential inconsistencies are in place to fix any governance issues.
- Track your cloud history - Collie CLI is built on the principles of GitOps and its data can easily be written to git repositories for tracking changes in your cloud environments.
The development is based on our in-depth experience with meshStack: Our multi-cloud governance platform for the enterprise. It provides you with an easy and lightweight entry to cloud governance.
Let's have a look into Collie and how it empowers you to manage your multi-cloud landscape!
Is your Cloud Landscape a Mess?
Did you ever get the feeling that you are losing track of what is happening in your cloud landscape? You have multiple environments (AWS accounts, Azure Subscriptions, or GCP projects) running across different clouds. They started as playgrounds and now more and more serious stuff is running in there, making it harder to keep track every day.
Your sheep are basically wandering off from the flock.
No matter which cloud you use: AWS, Azure, or GCP, they all provide capabilities on the org-level to see what cloud environments are running in there, and what costs they cause.
What is missing though is a comprehensive overview. It's all hidden behind different screens and settings, across multiple cloud consoles or CLIs, not allowing you to get that single picture you're looking for.
Cloud Transparency is Key!
For a proper overview, you would need to have different types of information, consolidated in a single view. It could be fairly easy, all you need are answers to these questions:
- Which cloud environments do exist in my multi-cloud landscape?
- Who do these cloud environments belong to?
- What costs do these cloud environments cause?
- Are they consistently tagged and documented?
Having this information at hand, enables you to bring things into order (= Cloud Governance) and herd your sheep.
Collie – Open-Source Multi-Cloud CLI Tool
This has never been as easy. All you need, is to have the native cloud CLI tools installed on your machine. From there it's just a couple of commands to shed light into the dark.
See for yourself:
If you are wondering how to use that information, here are some common use cases that people implement with Collie.
1) You want to delete cloud environments that aren't used anymore to reduce complexity.
2) You want to implement a consistent tagging strategy and need to know which cloud environments are still missing
3) You want to make sure that there are no idle environments (Sandboxes, etc.) raising your cloud bills every month
4) You want to implement a consistent tagging strategy and need to know which tags are currently used within your cloud environments
5) You want to map cloud environments to IT products, applications or cost centers within your organization
6) You want to analyze cloud costs based on their tags to build multi-cloud cost reports with nice dashboarding tools like PowerBI or Google Data Studio
How to get started with Collie
If you're looking for full transparency on your growing cloud landscape, Collie is just right for you.
Collie is open-source and available on GitHub. You only need to have the native cloud CLIs installed on your machine and are ready to go.
We hope you find it useful. Please reach out with any contributions, feedback, ideas or requests for help.
We're looking forward to herding our clouds together.
Why your ITSM Tool is the wrong Choice for cloud-native Cloud Management
ITSM tools have been used by large enterprises for ordering and workflow processes for a long time. They provide an easy way to order phones, monitors and other IT gadgets that could make an employee happy.
Providing Azure Subscriptions or AWS Accounts within your ITSM Tool
As cloud has become a new and common IT resource, it may be an obvious solution to use an existing ITSM tool to order specific cloud resources or entire cloud tenants.
A central IT service portal has the benefit that it consolidates all offered services that are used across the organization in a single place. Every employee knows that it exists and will probably use it as a first access point when it comes to IT requirements.
However, this general applicability is also its weakness, looking at highly specific services like cloud tenants. The cloud has much more complexity than a phone or a monitor. Here are just a couple of examples why:
- Access and Permission Management: The user circle of a cloud project will change and evolve over time (new team members, team members leaving the company).
- Cloud providers have heterogeneous organizational models. For example you have Management Groups in Azure and Service Control Policies in AWS. Both of these concepts can be used to group accounts within the organization and apply certain security guardrails to them.
- Cloud projects have a dynamic lifecycle. Not only users and resources change over time, governance guardrails may have to be updated eventually, which can require reconfiguration of accounts for example.
So while a workflow automation as provided by ITSM tools may help when it comes to the initial cloud tenant provisioning, it will hardly be able to cope with the complexity and required cloud governance that a large scale use of cloud by DevOps teams requires.
Building a Cloud Management Platform for Cloud Resources
Due to this complexity of cloud infrastructure, a continuously evolving cloud service offer and organizational implications resulting from the use of cloud (DevOps teams), the product category of cloud management platforms has evolved.
And it certainly makes sense to address cloud complexity with a tool that has been built for this specific purpose, which is what we at meshcloud do with our platform. Our tool enables organizations to be cloud-native. While we provide a framework for cloud governance, DevOps teams are able to use the native cloud tooling they are used to – no barriers, no abstractions. It's the organizational structure that we take care of: IAM integrations, cloud access management, tenant creation, policies and billing, not the workload.
But there is another aspect that should be considered:
This may sound a bit dramatic, but a lot of central IT departments have lost trust by their (internal) customers (DevOps teams) due to a lack of process adoption. They are associated with slow enterprise processes, legacy tooling and security restrictions that don't allow for any fun tech experiments or technological freedom.
This image problem brings IT departments into a difficult position to launch a new and isolated tool for cloud management. In order to gain back trust by the DevOps teams, they would have to market it internally to raise awareness ensure its adoption.
Therefore, it often makes sense to use an existing IT service portal as a central entry point to the (mesh)cloud world to increase the discoverability and avoid parallel activities in this field. Furthermore, there may be adjacent service offers, that better fit into such a service portal.
So, how to combine your IT Service Portal with a new Cloud Management Platform?
The central IT service portal stays the central access point for all IT services. However, you will not directly order cloud services like AWS accounts, Azure subscriptions or GCP projects within this portal. Instead, you will use it to get access to a cloud-native cloud management.
In the case of meshcloud as a cloud-native multi-cloud management tool, this process looks as follows:
Let's say you are DevOps team lead Theresa and you need to get cloud access for your DevOps team to develop this cool new application that runs in the cloud.
1) The central service portal is the initial entry point. The DevOps team lead Theresa can access it to request cloud access. Instead of providing her with individual cloud services, she will get access to meshcloud, her entry ticket to the cloud.
2 + 3) In some cases organizations want to include some approval workflows, which is something that workflow tools do pretty well. For example, Theresa may need to get a budget approved before actually getting access to the cloud. Another use case we often face is that cloud access requires some consultancy services, security trainings or a "cloud license" when used for the first time. These aspects can be covered in these two steps.
4) Once Theresas access has been approved, the central IT service portal will trigger the creation of a meshcloud account with the provided metadata. This process can easily be automated via our meshcloud API.
5) Now, our DevOps lead Theresa can access meshcloud to manage the cloud environments for her application. This includes permissions for other team members as well as the creation of cloud tenants in the available cloud platforms. In order to be able to map her activities in the cloud to the resulting application, she can provide some metadata, like an application ID, a cost center, a responsible product owner or a data security classification for the application.
6) The tenant creation and configuration is an automated process.
7) As this is a cloud-native portal, the DevOps team will have direct access to the cloud native consoles to set up and automate their deployments.
8) Cost management and chargeback can be a pain when managing a large number of projects in the cloud. Costs have to be mapped to their causing applications. meshcloud takes care of this cost allocation even across multiple clouds and provides cost data to be processed in further controlling and chargeback systems.
What's the Role of Central IT in this?
Right, we don't see central IT in this picture, but this doesn't mean they disappeared. So I want to spend the last words on common organizational structures we face in large enterprises. Within the central IT department, we often find a group of cloud experts (Enterprise Architects, Security and Governance, Platform Operators) – often called a cloud foundation. Their aim is to deliver secure cloud tenants to their internal customers (DevOps teams), to provide them with managed services like ELK stacks or databases and to assist them when it comes to security and compliance questions. meshcloud provides them with a tool, to do this in a consistent and scalable fashion.
To learn more about meshStack, please get in touch with our sales team or book a demo with one of our product experts. We're looking forward to get in touch with you.
(Multi-)Cloud Outlook in 2020
As meshcloud, we are proud to look back on a successful year 2019, in which we won multiple new customers. During this time we've had the great chance to accompany different types of companies in their cloud transformation. While our customers belong to many different industries, e.g. Automotive, Financial Services or Utilities, we've been able to observe many similarities in their move to the cloud – showing the relevance of cloud management and cloud governance among large organisations.
At meshcloud, we support large organisations in their cloud transformation. Our platform provides consistent governance with cross-platform Identity and Access Management, Tenant management, Cloud Security & Compliance and Cost Management and is used by large enterprises to deliver secure cloud environments to thousands of teams while staying in control and reducing complexity.
Based on our experiences in this past year, here are some predictions on upcoming cloud challenges in 2020:
Experiments come to an end: Bringing the Cloud Journey to the next level
A lot of companies have surpassed the experimental stage of using cloud. They are moving towards production deployments of cloud applications, have to manage rising costs and face tightening cloud security requirements. While in the beginning of a cloud journey, we often times face scattered cloud initiatives across different parts of the organisation, we now see more and more companies consolidating these initiatives and bringing them to the next level, setting up professional guardrails, preparing for wider user groups. One of the biggest challenges in this move is a reorganisation of responsibilities, especially in terms of cloud security, as the use now will surpass a couple of cloud experts.
In order to fulfill these requirements, there will be a consolidation of deviating cloud governance activities, bringing them together and solving them consistently across different cloud providers, often conducted by a dedicated team of (multi-)cloud experts. These new solutions have to be set up for scale in order to address the rising demand for cloud resources. Self-service and automation help to accelerate the time-to-cloud to a couple of minutes for increased productivity.
Watch out: Organizational lock-in is on the move
Being obsessed with avoiding vendor lock-in on the workload level a lot of companies have managed to transfer workload to infrastructure like Kubernetes and data to open-source databases like MySQL, PostgreSQL or MongoDB. At the same time the maturity of managed Kubernetes offers has greatly improved. However, as the use of cloud grows organisational processes and integrations towards cloud platforms are being established. And they are a new way for cloud providers to keep their customers close. As most organsations use more than one cloud provider to meet all their requirements, they are confronted with multiple organisational models. This is rather challenging, as the integration of new platforms becomes time and resource consuming and slows down the actual adoption of the new technologies.
We are very happy to be working on a declarative definition of organizations ("org-as-code") that enables companies to define their organization once, while meshcloud takes care of translating this into the different cloud platforms. The concept is known from infrastructure as code and based on the idea of defining a desired target state in an abstract way, while the system works out the specific implementations in the background. Having this, it becomes very easy to integrate new cloud platforms and provide technological freedom. (Learn more about org-as-code in our multi-cloud maturity model)
Cloud Security Complexity is bursting
When cloud is widely used across the organisation and the number of cloud projects is continuously growing, it is essential to have clearly defined cloud security concepts as well as proper process documentation to keep transparency on ongoing activities. Especially cloud access management (e.g. provisioning and retrieving access rights to cloud tenants) must be handled consistently, as the amount of managed permissions grows exponentially over time: more users, more projects, more cloud platforms. An easy and highly automated approach will help to keep control and enables to run continuous governance in order to comply with cloud security frameworks and certifications.
At the same time, multi-cloud teams will have to help cloud users and DevOps teams to implement cloud security concepts (e.g. via landing zones in their cloud tenants to achieve consistent security configurations across accounts and help them to deal with the increasing responsibility of DevOps practices.
Therefore, for 2020 we see an increased demand for clear governance of (multi-)cloud environments. Consistent cloud configurations and policy definitions will be key as companies scale up their use of cloud. Not only do they have to be controlled once implemented, their definition and scalable rollout will be in focus in order to provide a basic cloud security level for all cloud activity, which helps to align cloud configurations and accelerate basic cloud security assessments, even in regulated environments.
Are these topics relevant to you in 2020? At meshcloud, we provide best practices for multi-cloud management and multi-cloud governance, integrated into a powerful software platform. Feel free to reach out and get in touch with our multi-cloud experts or book a demo.
Why we're sponsoring the Dhall Language Project
We're very happy to announce that meshcloud is the first corporate sponsor of the Dhall Language Project via open collective. In this post I want to explain how we came to use Dhall at meshcloud, what challenges it solves for us and why we hope it will play a role in enabling software projects to more easily adapt to multi-cloud environments.
Enabling DevOps at scale
At the beginning of this year, we realized we had a challenge scaling configuration and operations of our software for customers. meshcloud helps enterprises become cloud-native organizations by enabling "DevOps at scale". Our tool helps hundreds or thousands of DevOps teams in an enterprise to provision and manage cloud environments like AWS Accounts or Azure Subscriptions for their projects while ensuring they are secured and monitored to the organization's standards.
Enabling DevOps teams with the shortest "time to cloud" possible involves the whole organization. Our product serves DevOps teams, IT Governance, Controlling and IT Management in large enterprises. That means meshcloud is an integration solution for a lot of things, so we need to be highly configurable.
Because we also manage private clouds (OpenStack, Cloud Foundry, OpenShift etc.) we often run on-premises and operate our software as a managed service. This presents unique challenges for our SRE team. Not only do we need to maintain and evolve configuration for our growing number of customers, but we also need to support deploying our own software on different infrastructures like OpenStack, AWS or Azure[1].
At the end of the day, this boils down to having good and scalable configuration management. After going through various stages of slinging around YAML with ever more advanced tricks, we realized we needed a more fundamental solution to really crack this challenge.
Configuration management at scale - powered by dhall
The Dhall configuration language solves exactly this problem. It\'s a programmable configuration language that was built to express configuration - and just that. Dhall is decidedly not turing complete. It\'s functional nature makes configuration easy to compose from a set of well-defined operations and ensures that configuration stays consistent.
Using Dhall allows us to compile and type check[2] all our configuration for all our customers before rolling things out. We use Dhall to compile everything we need to configure and deploy our software for a customer: Terraform, Ansible, Kubernetes templates, Spring Boot Config. We even use Dhall to automatically generate Concourse CI pipelines for continuous delivery of our product to customers.
Since adopting Dhall earlier this year, we measurably reduced our deployment defect rate. We feel more confident about configuration changes and can safely express configuration that affects multiple services in our software.
Empowering a Multi-Cloud Ecosystem
We believe that open-source software and open-source cloud platforms are crucial for enabling organizations to avoid vendor lock-in. Now that mature tools like Kubernetes exist and can do the heavy lifting, enabling portability between has become a configuration management challenge.
What we found especially interesting about Dhall is that it\'s not just an "incremental" innovation atop of existing configuration languages like template generators, but instead looks at the problem from a new angle and tries to solve it at a more fundamental level. This is something we can relate to very well as we\'re trying to solve multi-cloud management using an organization as code (like infrastructure as code) approach.
That's why we\'re happy to see Dhall innovating in this space and reached out to the Dhall community to explore ways we can support the project. We hope that providing a steady financial contribution will allow the community to further evolve the language, tooling and its ecosystem.
Footnotes:
-
[1]: In this way meshcloud is not only a multi-cloud management software but is also a multi-cloud enabled software itself.
-
[2]: Dhall purists will want to point out that expressions are not compiled, instead they\'re normalized.
Multi-Clouds in Banking – 3 Takeaways from the 2nd EBF Cloud Banking Conference
It’s already been a while. On July 9th, I visited the 2nd EBF Cloud Banking Forum in Brussels. The topic of the day: “Shaping a Multi-Cloud Environment”. Representatives of banks, regulators and cloud providers met to discuss how cloud computing can be used in the financial services sector, with one common goal in mind: To provide secure banking services to the people. With all the FinTechs and Neobanks showing up all over the place, it is obvious that traditional banks have to adopt new technologies to stay competitive in the market. The EBF Cloud Banking forum brings this discussion to the European level. With joint forces, we can be an example not only to other countries, but also be a pioneer in regard to other sectors.
The adoption of cloud technologies comes with some hurdles and quickly raises questions for the users: “How do I stay compliant with existing regulations?” as well as the regulators: “How do I make clear and transparent rules?”. “How are we going to treat the overcententration on the market?” There basically are only a handful of cloud providers to choose from and implementing a multi-cloud strategy is one way to act against overconcentration. However, it also raises the administrative complexity of adopting cloud. Therefore, a proper cloud management has to be put in place to help the banks avoid having more complexity than before. From a whole day of intensive discussions, I’ve put together the three main take-aways of the day.
Identity and access management (IAM) is crucial to a bank’s cloud transformation
This topic is not new to external environments like clouds. Defining how to manage identities and access to infrastructure resources is and has been a complex issue for many years. In order to mitigate the risk of abuse or unauthorized access to confidential information and critical systems, access rights have to be reduced to a minimum, following the principle of least privilege. Having a multi-cloud environment may raise the complexity of achieving this.
On the one hand, you want to make it easy for your users to access cloud resources, in order to speed up software development and lower the barrier for cloud adoption. On the other hand, you need to be in control of the access process, make sure it is documented and auditable to avoid the risk of undiscovered information leakage.
With meshcloud, we help large organizations like banks with a governance framework for multi-cloud environments. Our platform plugs into the existing organizational structures and identity providers like Azure AD or LDAP and unifies the way access and permissions are handled across different cloud providers. As a result, we use existing identities and provide secure SSO access to all cloud providers as well as a platform to manage fine-granular permissions centrally before replicating them to the attached platforms.
In addition, cloud providers like AWS and Azure have published services like AWS CloudTrail that monitor access and operations related to your cloud infrastructure.
An exit strategy is necessary to avoid vendor lock-in to a single cloud provider
The EBA guidelines on outsourcing arrangements include a paragraph on exit strategies, which requires banks and other financial institutions to have a well-documented exit strategy in place that ensures business continuity in case of an interruption of outsourced services. This can be the result of a failure, malfunctioning service offers as well as contractual or legal obstacles in regard to the service provider. These scenarios should be well documented and practiced, as it is crucial to be familiar with the workflows and know what to do in such exceptional circumstances.
While multi-cloud is a way to distribute the risk across more than one service provider, there are strategic steps to be considered when deciding on a sustainable cloud strategy. The use of open-source infrastructure components on different layers, such as Kubernetes, PostgreSQL or NGINX for example, can facilitate migrations from one cloud to another drastically. They can even be operated in a private datacenter. However, most companies decide on a multi-cloud strategy as a result of a best-of-breed approach. They want to use each cloud for the specific (and often proprietary) services it offers, for example in the field of machine learning or artificial intelligence. The use of such tools can accelerate development because they bring a lot of functionality out-of-the box and enable teams to focus on truly differentiating functionality. An assessment of the criticality of applications as well as the consequences of the failure of such proprietary services should be considered within an exit strategy.
As meshcloud, we help companies to avoid vendor lock-in by facilitating the use of multiple cloud platforms and the distribution of infrastructure and applications across them. We define organizational structures (teams, projects, users, permissions) in a declarative manner to enable our customers to integrate new service providers very fast and with little administrative overhead. To address the rise of higher-level infrastructure services, such as managed databases, message queues or machine learning and AI services, we offer a service marketplace that enables customers to provide a large variety of services based on an open standard (OSB API). This unifies the process of provisioning services and decouples it from the underlying infrastructure.
Configuration is key
A recent study of KPMG found that 80% of IT security incidents happen due to manual misconfigurations. By spreading infrastructure across different service providers it gets even harder to keep control on correctness and consistency of cloud configurations and this does not consider any application-specific configurations yet.
To improve security and compliance of cloud environments, our landing zones help to configure cloud accounts upon creation. This enables our customers to roll-out consistent configurations in automatically created cloud accounts, according to their use case (test, development, production). To give an example these configurations can limit the use of cloud infrastructure to specific geographical regions or blacklist certain services that are not compliant. By rolling them out consistently across all cloud accounts, our customers can relieve their development team from defining compliant configurations individually and instead provide them with a framework that has been approved by the security department.
To learn more about meshStack, please get in touch with our sales team or book a demo with one of our product experts. We're looking forward to get in touch with you.
The path to proper multi-cloud management – a maturity model
Cloud means Multi-Cloud
The cloud is the foundation of an efficient IT operation in the digital age. More often than not, one cloud platform is not sufficient for at least two reasons: 1) Different cloud platforms service different purposes and each has specific strengths and weaknesses hence embracing the plurality of platforms in a best-of-bread manner is more productive than trying to fight it. And 2) To avoid a technology lock-in on one vendor or platform, creating a diverse skill set preserves technical sovereignty.
Therefore, building a proper multi-cloud management capability is key to a successful and efficient cloud strategy. To give some guidance about the technical approaches to build such a capability, we introduce the meshcloud multi-cloud management maturity model.
The meshcloud multi-cloud management maturity model
Stage 0: No Multi-Cloud Management
The first stage is easy to reach – it is the natural stage when you do not take any actions, maybe because you do not realize you have a multi-cloud challenge to solve. However, the effects rise nevertheless.
Due to disorientation of users individual platform or ad-hoc processes will be established and you end up with a silo-structured organization loaded with unnecessary complexity due to different access routines for different platforms.
But that is not necessarily the greatest worry: people tend to shortcut complexity and lengthy processes and public cloud vendors make it way to easy to access cloud resources within minutes with the simple (yet virtual) swipe of a company credit card. The nightmare has a name: Shadow IT! You end up with individual, unconnected, unregistered public cloud accounts the company pays for but has no efficient way to manage. All comes down to individual initiative and discipline which might go well, but it’s an anarchy and there is no way to ensure rules and regulations are followed and compliance is uphold. This has severe implications for security or regulatory duties, e.g. you cannot give full report about data processing activities as required by the GDPR. Accounts might be shared among colleagues. When people leave the company, no one guarantuees credentials are rotated. And so on.
Therefore, this stage is like purgatory – you suffer from sins of the past and want to get out as soon as possible. To receive remission, action is necessary.
Stage 1: Manual / Semi-automated Multi-Cloud Management
So we realize: let’s bring order to the chaos. We try to standardize, define common processes, keep track of tenants and workloads out there in the clouds or running within our own private cloud.
In Stage 1, we define a process to provision and configure tenants on the clouds we use. Someone is defined responsible for a certain platform, to keep security and governance up and to provide users with what they need to get their cloud projects up and running.
Regarding implementation, we move quick and take the tools we have and are familiar with: a ticketing system to order cloud tenants from the IT department which someone creates manually or provides manually or with some scripting on her or his own. What about controlling? Let’s keep track of cloud tenants in an Excel sheet or export a list of projects on a platform to see who’s doing what and how much of it.
It’s tedious and everyone hates it – those who want to use the cloud because there still different contacts for different clouds and because they have to wait so long to receive their credentials. And operators hate it because they are doing dull work and lots of it instead of improving processes and evaluating further advanced technologies.
The cloud is all about automation and the multi-cloud is, too! So this stage does not fulfill the promise of self-serving, on-demand, scaling capacities with a few key strokes, but falls behind the spirit that comes with cloud technologies: automate ALL THINGS.
Stage 2: Imperative / Workflow-based Multi-Cloud Management
With further investment, we reach stage 2 where we have reached the stage of integrated, automated processes to provision tenants and user accounts in all relevant cloud platforms. We might even have established a self-service where users are able to set up cloud projects 24/7 at their own pace and needs. There is a portal where they can order cloud projects or ressources and after a while they are ready to run.
So all good now? At first, it might be. But over time, workflows and tenant configurations evolve and after some months or years it becomes hard to tell which tenants were created and configured by which version of the process. Also, do people reliably remove resources when they are not needed anymore? And how are resources in different clouds connected and how do they depend on each other? Hopefully, the documentation is available and up to date – otherwise, we still lose orientation between platforms and projects consuming and possibly spanning them.
But what now?
Stage 3: Declarative Multi-Cloud Management
Meet declarative multi-cloud management! You already know the concept from the deployments of your workloads where it labels as infrastructure as code. You describe the desired target state and the system works it out for you. Declarative multi-cloud management means the same, but on an organizational, cross-platform level – Organization as Code as we framed it.
So you define which projects should exist on which platform and which users should have what kind of access. You save the new configuration in a versioned repository and after a short while the whole multi-cloud federation converges to the new state you described. The version control enables you to frantically audit every single change that happened to your multi-cloud landscape. This has several advantages:
- Robustness: Distributed systems might suffer from weak network links which might cause process flows to stall and fail. However, with the target state clearly defined, the system can try again and again to finally reach the desired state.
- Prevention of local changes: You know the “colleague is on vacation, but I need urgent access” situation which might lead to a quick and dirty permission to cloud projects by an administrator. More often than not, those changes stick longer than intended – the human lazyness is my witness. However, a declarative multi-cloud management system would roll back those changes at the next sync.
- Decoupling of semantics and implementation logic: With a descriptive configuration you describe what you want, not how to do it. Therefore, flexibility remains with the system on how to implement reaching the desired state which means more freedom to improve the multi-cloud system over time.
- Documentation and Auditing: If every configuration change is tracked in the manifest and version controlled, it is always easy to report, audit and track who had access to which system or data and how long, why and by whom.
You see that declarative multi-cloud management has many advantages – it’s for some reason the paradigm turned so successful with configuration management. Therefore, meshcloud put the declarative paradigm at the heart of its vision of an agile, but full-control multi-cloud management. If you want to learn more, we are happy to start the conversation with you.
Reach out to us via mail, social webs, phone or any events we attend. We are looking forward to hear your thoughts!
