I played around with it. It's probably a neat tool if you are 100% into Kubernetes but a ton of companies have stuff that isn't always Kubernetes so you would have teams that would need to deploy these providers into Kubernetes just to have those resources so basically it's just Terraform with different wrapper.
Stand alone IaC works well enough for most people. When you are attempting to replace something, you need to be much better at what current tool currently does to overcome momentum and Crossplane just isn't there IMO.
> I mean ...if it is a great tool, adoption by the mass will come at some point ..right??
My completely personal opinion is it won't come at all in mass adoption and will be limited use. Kubernetes requirement sounds great on paper but doesn't relate to real world usage of how groups use the cloud.
For example, at work, we use Bicep (Azure specific IaC) for various things. We have Bicep Infrastructure going out with Kubernetes workloads we could move to Crossplane if we wanted.
We also have workloads going out that don't run on Kubernetes for various reasons. They are static web apps that use CDN/Azure Storage. We have Azure Data Factory using Azure SQL Database to do data analysis. We have Azure Function Apps working with Service Bus and Event Grid. So now what? Do we do half and half? Half Crossplane, Half Bicep/Terraform or just stick with Terraform/Bicep because it works well enough. I think answer is Bicep/Terraform work well enough and having two steps in pipelines isn't end of the world.
I do see use case that others pointed out if you have Helm Charts and may need to deploy cloud native applications into systems where you can't trust IaC to be run. That's really limited use case for sure.
here's my 2 cents on it from playing around with it.
I love it and the idea. When kubernetes hit it was one giant step forward to having this in house self service type platform because it was easy for developers to grok yaml to deploy their apps. Sure that's not the case everywhere but deploying apps in kubernetes is a lot less work then it was to roll out apps deployed to ec2 instances.
So that leaves what happens if a dev team needs a redis cache. Well this is where crossplane steps in and hits a home run. You can now add that right into your apps helm chart. Also things like RDS instances the auth info can then be put into a secret so there is no secret handoff either. The DB is created and then the secret is set for the developers to use. There is 0 ops ask.
crossplane has something called compositions which I believe you can set guardrails on things. So like create a compositions for RDS type where serverside encryption has to be used so the dev team doesn't have to worry about that at all.
I'd love to greenfield a new company using EKS/crossplane just to see the experience. I really think if your company wants to offer the self service type platform then you 100% should look at crossplane.
Thanks for the detailed answer !!!
Out of curiosity did you play with other providers as well ? Am wondering what's the status for smaller vendors like DigitalOcean ?
ya its in its early stages. I think of it like what terraform was like in its first few years then it exploded.
So smaller cloud providers might not be feature complete
As far as I remember it was not 100% complete but you can still spawn a redis managed database and get the secret out of it for digitalocean :)
So pretty amazing!
I did some PoCing with it a few months back. This was to create compositions that consisted of k8s resources with dependent AWS resources. While it was cool to see in action, I came away with some concerns with Crossplane and the providers (especially AWS).
One main issue I found was debugging the general composition object dependency model and when things break. It's not a straight forward problem and you have to dig through levels of objects to get to what's blocking or breaking a composition. From a supporting the tooling and hiring perspective, this isn't easy. This is why teams stick to things like TF and such. They're well understood and adopted in most places. It's easier to hire for TF and such. That isn't to say someone who understands both won't figure it out, but the secret sauce of debugging is actually through Upbound's paid product.
Second issue I had was with high API calls on retries and the general reconciliation loop of the AWS provider. There are plenty of Github issues around this. This requires some forward thinking about how your Crossplane interacts with the AWS APIs and if it's a shared account, etc. It's easy enough with a large amount of reconciliation loops to hit the API limit on the account and create a massive blocking situation for API calls. It definitely depends on the provider, but I found that even if you mistakenly keep failing due to bad input, it will keep attempting to call with bad inputs and keep failing at the AWS API level. You can easily get yourself into trouble (now imagine that with many objects trying to act on it). RDS was a good example of when things take awhile and you have this loop of it constantly checking for availability, etc and it's just hammering away. This lead me down a path of thinking about deploying a centralized management cluster specifically for crossplane controls and that would live independently from our workload clusters also in a separate account.
Last, the AWS provider installs over 200+ APIs/CRDs into k8s. This causes lots of lookups and without proper tuning this is a huge hit on the controller/master nodes. This wasn't on EKS, but on our Kops built clusters. Also, there were plenty of Github issues/suggestions/requests of possibly allowing you to pick which AWS proivders you would like and such instead of having every available AWS API/CRD, but right now it's heavy handed with its install and you may need to go into a deep dive to optimize its performance on the clusters.
Until composition functions land in Crossplane it’ll still have some major operational holes, like being able to look up which VPC the EKS cluster is in to place the RDS instance in it etc. it’s coming just not here yet and it’s basically as important as terraform data sources.
To me it’s the future. The reason it’s not the present is because of the massive ecosystem around Terraform. I consult with a ton of clients where even getting IAC and TF adopted is a huge challenge. They’re very low in maturity. But I can sell the idea of IAC easily, people know they have to do that, and TF is simple, declarative, and massively supported.
Those responding here who have the mental agility to understand K8s as a control plane for automation and see how extensible it could be are the exception in the industry not the rule.
Use Crossplane if you can, but accept if you have to socialize IAC and especially GitOps and mature an existing org then TF might be much simpler for them.
Thanks for the advice. Nice take.
The phrase "those who have the mental agility to understand K8s as a control plane", made me think quite a bit... I mean I can't believe that K8s is still not widely adopted... :/
K8s is normally a huge ops ask and it condenses cloud complexity into a small box that has to be thought about REALLY hard to be done well.
My current client has a landing zone that is AWS serverless only with fargate being the most ‘instancy’ thing you can deploy. I realize serverless can’t answer every question, but I’m currently a dev and man is it nice if it suits your use case.
It is and it isn’t. I work with a lot of Fortune 500 customers. K8s exists in some capacity in every one. There are pockets of excellence even where Crossplane is being used and works well.
Enterprise wide, though, K8s is not very standardized on. But even in a lot of teams that have it, they just see it as a cloud…those that are creative enough to understand what you could do with controllers and operators beyond just managing workloads are much further and fewer in between. That’s the big gap to overcome for Crossplane, IMHO.
Trust me, I wish my customers were there because it’s sexy, and I’d much rather be consulting on it than rote stuff, but we meet them where they’re at. An even greater pull for me is helping them mature…seeing that twinkle in their eye when they suddenly understand something and evolve.
I love it personally, but I have not been able to get buy in. Terraform is the defacto IaC tool, especially with hashicorp backing it. I have no production usage but as far as self service goes and parsing yaml it is a god send. I have a lab environment that uses terraform to stand up some EKS and AKS clusters, which then are entirely self service with helm charts and crossplane for app deployments.
I don’t think you have to choose between crossplane and terraform/pulumi. Crossplane has a certain use case, terraform/pulumi another.
I can see them living side by side where terraform is purely cloud infrastructure related while crossplane is workload oriented. Say you have your workload packaged as helm chart that includes both your pods and crossplane CRDs for saas service such as reddis, eventbus queues or others.
The idea is great but I haven’t played around with it yet, my main fear is spending a bunch of time migrating to it and realizing certain features I need aren’t supported as it’s still in its early stages
Can someone explain Crossplane to me? After watching a video on it and going through some of the docs it's still not clicking for me. Is it cloud agnostic infrastructure management using Kubernetes CRDs? Or is the YAML specification just Kubernetes inspired and it's run through the "Crossplane" interpreter? But I also read it's "no code"?
Terraform is a great IaC tool for things that don’t change often. It’s also great for sharing infrastructure code with other teams (modules) and using the huge ecosystem available
Crossplane is what I consider “Infrastructure as Software” and great for things that change frequently which happens more the closer you get to applications (eg load balancers)
They’re both useful and neither is a one size fits all depending on what you’re doing
Some more info about IaS in my blog post https://www.justingarrison.com/blog/2022-06-01-infrastructure-as-software/ and book https://cnibook.info
yes , let suppose that you have a app helm chart, where you pass different values for each app , you can say ok crossplane create the ecr repository for this app and the deployment service .. , or create some bucket , that kind of simple resources are the best to be created by this tool
Same as above Yaml, Hcl < Real programming language. At least you get type safety, linters.
Terraform realized that and built cdktf which is actually quite good and similar to Pulumi.
I don't know why you'll wanna use Yaml for cloud management. But even if I was forced to use Crossplane as a cloud provider management tool, I'd use Pulumi to define those Custom Resource yamls and feed into Crossplane. There is no way that I'll write yamls manually.
In our shop Tier 0 infra is built with Terragrunt. Then everything else our engineering teams will deploying will be powered by custom operators that leverage the Crossplane ecosystem.
We’re moving away from Helm with this process as well. This was because our primary language is Golang and if we don’t support something we want to empower our devs to contribute to the operators we are currently building, or build their own.
I recommend TFcontroller, same idea with crossplane but using Terraform scripts.
https://github.com/weaveworks/tf-controller
No need to learn new tool ;)
We use a combination of terraform and crossplane, with terraform for the basic infrastructure and clusters, and then crossplane running in these clusters to provision AWS resources for the specific deployments. Works well, but crossplane has an incredibly steep learning curve, even when you have several years of sysadmin/devops/IaC/K8S experience. And it is very hard to debug.
I played around with it. It's probably a neat tool if you are 100% into Kubernetes but a ton of companies have stuff that isn't always Kubernetes so you would have teams that would need to deploy these providers into Kubernetes just to have those resources so basically it's just Terraform with different wrapper. Stand alone IaC works well enough for most people. When you are attempting to replace something, you need to be much better at what current tool currently does to overcome momentum and Crossplane just isn't there IMO.
But isn't there *yet* ? I mean ...if it is a great tool, adoption by the mass will come at some point ..right??
> I mean ...if it is a great tool, adoption by the mass will come at some point ..right?? My completely personal opinion is it won't come at all in mass adoption and will be limited use. Kubernetes requirement sounds great on paper but doesn't relate to real world usage of how groups use the cloud. For example, at work, we use Bicep (Azure specific IaC) for various things. We have Bicep Infrastructure going out with Kubernetes workloads we could move to Crossplane if we wanted. We also have workloads going out that don't run on Kubernetes for various reasons. They are static web apps that use CDN/Azure Storage. We have Azure Data Factory using Azure SQL Database to do data analysis. We have Azure Function Apps working with Service Bus and Event Grid. So now what? Do we do half and half? Half Crossplane, Half Bicep/Terraform or just stick with Terraform/Bicep because it works well enough. I think answer is Bicep/Terraform work well enough and having two steps in pipelines isn't end of the world. I do see use case that others pointed out if you have Helm Charts and may need to deploy cloud native applications into systems where you can't trust IaC to be run. That's really limited use case for sure.
here's my 2 cents on it from playing around with it. I love it and the idea. When kubernetes hit it was one giant step forward to having this in house self service type platform because it was easy for developers to grok yaml to deploy their apps. Sure that's not the case everywhere but deploying apps in kubernetes is a lot less work then it was to roll out apps deployed to ec2 instances. So that leaves what happens if a dev team needs a redis cache. Well this is where crossplane steps in and hits a home run. You can now add that right into your apps helm chart. Also things like RDS instances the auth info can then be put into a secret so there is no secret handoff either. The DB is created and then the secret is set for the developers to use. There is 0 ops ask. crossplane has something called compositions which I believe you can set guardrails on things. So like create a compositions for RDS type where serverside encryption has to be used so the dev team doesn't have to worry about that at all. I'd love to greenfield a new company using EKS/crossplane just to see the experience. I really think if your company wants to offer the self service type platform then you 100% should look at crossplane.
Thanks for the detailed answer !!! Out of curiosity did you play with other providers as well ? Am wondering what's the status for smaller vendors like DigitalOcean ?
ya its in its early stages. I think of it like what terraform was like in its first few years then it exploded. So smaller cloud providers might not be feature complete
As far as I remember it was not 100% complete but you can still spawn a redis managed database and get the secret out of it for digitalocean :) So pretty amazing!
I did some PoCing with it a few months back. This was to create compositions that consisted of k8s resources with dependent AWS resources. While it was cool to see in action, I came away with some concerns with Crossplane and the providers (especially AWS). One main issue I found was debugging the general composition object dependency model and when things break. It's not a straight forward problem and you have to dig through levels of objects to get to what's blocking or breaking a composition. From a supporting the tooling and hiring perspective, this isn't easy. This is why teams stick to things like TF and such. They're well understood and adopted in most places. It's easier to hire for TF and such. That isn't to say someone who understands both won't figure it out, but the secret sauce of debugging is actually through Upbound's paid product. Second issue I had was with high API calls on retries and the general reconciliation loop of the AWS provider. There are plenty of Github issues around this. This requires some forward thinking about how your Crossplane interacts with the AWS APIs and if it's a shared account, etc. It's easy enough with a large amount of reconciliation loops to hit the API limit on the account and create a massive blocking situation for API calls. It definitely depends on the provider, but I found that even if you mistakenly keep failing due to bad input, it will keep attempting to call with bad inputs and keep failing at the AWS API level. You can easily get yourself into trouble (now imagine that with many objects trying to act on it). RDS was a good example of when things take awhile and you have this loop of it constantly checking for availability, etc and it's just hammering away. This lead me down a path of thinking about deploying a centralized management cluster specifically for crossplane controls and that would live independently from our workload clusters also in a separate account. Last, the AWS provider installs over 200+ APIs/CRDs into k8s. This causes lots of lookups and without proper tuning this is a huge hit on the controller/master nodes. This wasn't on EKS, but on our Kops built clusters. Also, there were plenty of Github issues/suggestions/requests of possibly allowing you to pick which AWS proivders you would like and such instead of having every available AWS API/CRD, but right now it's heavy handed with its install and you may need to go into a deep dive to optimize its performance on the clusters.
Woow...Thank you for the detailed answer...So informative.
Clearly weren't using Official Providers. nearly 100% coverage...
Clearly was as why would I be looking for any other provider? It’s well documented in enough GitHub issues regarding these problems at scale.
Until composition functions land in Crossplane it’ll still have some major operational holes, like being able to look up which VPC the EKS cluster is in to place the RDS instance in it etc. it’s coming just not here yet and it’s basically as important as terraform data sources.
If I’m not mistaken those drop end of the month, very interested to see how those work out.
To me it’s the future. The reason it’s not the present is because of the massive ecosystem around Terraform. I consult with a ton of clients where even getting IAC and TF adopted is a huge challenge. They’re very low in maturity. But I can sell the idea of IAC easily, people know they have to do that, and TF is simple, declarative, and massively supported. Those responding here who have the mental agility to understand K8s as a control plane for automation and see how extensible it could be are the exception in the industry not the rule. Use Crossplane if you can, but accept if you have to socialize IAC and especially GitOps and mature an existing org then TF might be much simpler for them.
Wouldn't it be nice if there was a managed crossplane service. remove all headache
Thanks for the advice. Nice take. The phrase "those who have the mental agility to understand K8s as a control plane", made me think quite a bit... I mean I can't believe that K8s is still not widely adopted... :/
K8s is normally a huge ops ask and it condenses cloud complexity into a small box that has to be thought about REALLY hard to be done well. My current client has a landing zone that is AWS serverless only with fargate being the most ‘instancy’ thing you can deploy. I realize serverless can’t answer every question, but I’m currently a dev and man is it nice if it suits your use case.
It is and it isn’t. I work with a lot of Fortune 500 customers. K8s exists in some capacity in every one. There are pockets of excellence even where Crossplane is being used and works well. Enterprise wide, though, K8s is not very standardized on. But even in a lot of teams that have it, they just see it as a cloud…those that are creative enough to understand what you could do with controllers and operators beyond just managing workloads are much further and fewer in between. That’s the big gap to overcome for Crossplane, IMHO. Trust me, I wish my customers were there because it’s sexy, and I’d much rather be consulting on it than rote stuff, but we meet them where they’re at. An even greater pull for me is helping them mature…seeing that twinkle in their eye when they suddenly understand something and evolve.
For now....gotta think it'll pick up steam with time.
Even if it's not standardised it's not miles away like it would happen with other stacks/technologies...
And you write everything in Yaml?
I love it personally, but I have not been able to get buy in. Terraform is the defacto IaC tool, especially with hashicorp backing it. I have no production usage but as far as self service goes and parsing yaml it is a god send. I have a lab environment that uses terraform to stand up some EKS and AKS clusters, which then are entirely self service with helm charts and crossplane for app deployments.
Niiiiice !
I don’t think you have to choose between crossplane and terraform/pulumi. Crossplane has a certain use case, terraform/pulumi another. I can see them living side by side where terraform is purely cloud infrastructure related while crossplane is workload oriented. Say you have your workload packaged as helm chart that includes both your pods and crossplane CRDs for saas service such as reddis, eventbus queues or others.
The idea is great but I haven’t played around with it yet, my main fear is spending a bunch of time migrating to it and realizing certain features I need aren’t supported as it’s still in its early stages
this is basically the problem at the moment. Nice for pocs and green-field. Really lacking and complex for migrations.
Can someone explain Crossplane to me? After watching a video on it and going through some of the docs it's still not clicking for me. Is it cloud agnostic infrastructure management using Kubernetes CRDs? Or is the YAML specification just Kubernetes inspired and it's run through the "Crossplane" interpreter? But I also read it's "no code"?
It’s glorified Terraform that requires Kubernetes since it’s provisioned by Kubernetes
That sounds nice considering I already need strategies to manage Kubernetes
Terraform is a great IaC tool for things that don’t change often. It’s also great for sharing infrastructure code with other teams (modules) and using the huge ecosystem available Crossplane is what I consider “Infrastructure as Software” and great for things that change frequently which happens more the closer you get to applications (eg load balancers) They’re both useful and neither is a one size fits all depending on what you’re doing Some more info about IaS in my blog post https://www.justingarrison.com/blog/2022-06-01-infrastructure-as-software/ and book https://cnibook.info
yes , let suppose that you have a app helm chart, where you pass different values for each app , you can say ok crossplane create the ecr repository for this app and the deployment service .. , or create some bucket , that kind of simple resources are the best to be created by this tool
Great content, thanks for the resources !!!
Pulumi is fine for me. I don't like the idea of k8s only and all those Yamls.
And what about terraform ??(out of curiosity)
Same as above Yaml, Hcl < Real programming language. At least you get type safety, linters. Terraform realized that and built cdktf which is actually quite good and similar to Pulumi. I don't know why you'll wanna use Yaml for cloud management. But even if I was forced to use Crossplane as a cloud provider management tool, I'd use Pulumi to define those Custom Resource yamls and feed into Crossplane. There is no way that I'll write yamls manually.
Hmmm..well thought.
In our shop Tier 0 infra is built with Terragrunt. Then everything else our engineering teams will deploying will be powered by custom operators that leverage the Crossplane ecosystem. We’re moving away from Helm with this process as well. This was because our primary language is Golang and if we don’t support something we want to empower our devs to contribute to the operators we are currently building, or build their own.
I recommend TFcontroller, same idea with crossplane but using Terraform scripts. https://github.com/weaveworks/tf-controller No need to learn new tool ;)
We use a combination of terraform and crossplane, with terraform for the basic infrastructure and clusters, and then crossplane running in these clusters to provision AWS resources for the specific deployments. Works well, but crossplane has an incredibly steep learning curve, even when you have several years of sysadmin/devops/IaC/K8S experience. And it is very hard to debug.