Terraform for data people: Infrastructure as Code in Microsoft Fabric
What Infrastructure as Code is and how Terraform works (HCL, providers, state and plan), explained with SQL and Delta Lake analogies for Microsoft Fabric.
deployment_steps.docx. And the _v2. And the _FINAL. And the _FINAL_FINAL2.
Inside: 27 screenshots of a version of the portal that no longer exists, screenshot 14 in English and the rest in Spanish, and one step only Jose knows how to do. And Jose is on holiday.
You’ve probably got one like it in a folder somewhere. The real problem isn’t the Word file: it’s that setting up an environment depends on a manual procedure that’s hard to repeat and that nobody keeps up to date.
At NetCoreConf, Alfonso Ming and I gave a talk called «Y yo aquí creando workspaces a mano» («And here I am, creating workspaces by hand») about how to stop doing this with Terraform on Microsoft Fabric. This is the theory, the things I wish I’d had clear before writing my first line. The hands-on part, with the demo video, is in the next article.
Let’s do the maths
A normal project: four environments (dev, test, QA and prod) and three medallion layers (bronze, silver and gold). That’s twelve workspaces before you’ve created a single lakehouse. With four roles per workspace, 48 permission assignments.
That’s an afternoon of clicking, and it’s very hard to get to the end without missing a role somewhere.
What happens with ClickOps
This is how most of us do it: you open the portal, configure things, take screenshots “to document it later”, and it works. The clicks aren’t the problem. The problem comes afterwards.
Three months later someone asks why the QA workspace doesn’t have the same role as prod, and nobody knows. To answer that you’d need to know what was there before, who changed it and why, and the portal doesn’t make that easy. If tomorrow you need the same environment for another project, you depend on the memory of whoever built it the first time, and two people following the same Word file end up with two different setups. And when an audit asks who has access to what, you’re rebuilding it by hand.
Keeping the configuration as code in Git solves a good part of this: changes get reviewed in a pull request and you have a history. One caveat: Git tells you what changed in the code and who approved it; who applied it and when, you’ll only know if the apply runs from a pipeline rather than from each person’s laptop.
What Infrastructure as Code (IaC) is
Infrastructure as Code means describing your platform in text files, keeping them in Git and letting a tool apply them for you. The files can be read, compared and copied from one project to the next, and Git gives you history, branches, pull requests and a way back. If Git still scares you a little, here’s my Git translator for data analysts.
Imperative or declarative: you already know this
There are two ways to automate. The imperative way is telling the machine how to do it, step by step, in a script where you write every if:
$ws = Get-Workspace -Name "Bronze-DEV"
if ($null -eq $ws) {
$ws = New-Workspace -Name "Bronze-DEV"
Set-Capacity -Id $cap -Workspace $ws
} else {
if ($ws.Capacity -ne $cap) { Set-Capacity -Id $cap -Workspace $ws }
}Does it already exist? Does it have the right capacity? You handle every one of those cases.
The declarative way is describing how you want it to end up, and the engine works out what it has to do to get there:
resource "fabric_workspace" "bronze_dev" {
display_name = "Bronze-DEV"
capacity_id = var.capacity_id
}If you work with data, you already know this: SQL is declarative too. You write the SELECT you want and you don’t tell the engine whether to use a hash join or a nested loop. Terraform does the same with the platform.
I’ve always found it odd that our table schemas have lived in code for years and the platform that holds them hasn’t.
The change in mindset
Once you adopt Terraform, the repo becomes the reference for how the platform should look, and the portal is what you get from applying it. You can still change something by hand in the portal, but that change sits outside the code and Terraform will try to undo it next time (more on that below, with drift).
What Terraform is
Terraform has three pieces. A CLI, the terraform executable, where you’ll mostly use init, plan, apply and destroy. A language, HCL: .tf files with blocks and key = value pairs, fairly easy to read even if you’ve never seen it. And providers, the plugins that know how to talk to each service.
On the licence: Terraform is free to use, but since version 1.6, in August 2023, it’s distributed under the BSL licence, which isn’t open source. If that matters to you, OpenTofu is the open fork and understands the same code.
A provider is a connector
Power BI doesn’t know how to talk to SAP; the SAP connector does. Terraform’s the same: it doesn’t know how to talk to Fabric, and the Fabric provider does. Your .tf code describes what you want, Terraform calculates the plan, the provider turns it into calls to the Fabric REST API, and the API does the work.
In Fabric you’ll use three:
| Provider | What for |
|---|---|
microsoft/fabric | Workspaces, roles, lakehouses, warehouses, notebooks, pipelines, domains, Git integration and more |
hashicorp/azuread | Entra ID security groups |
hashicorp/azurerm | The Fabric capacity and the storage account where you keep the state |
They work together fine: in the same file you create a group in Entra ID and assign it to a workspace.
Anatomy of a resource
resource "fabric_workspace" "bronze_dev" {
display_name = "Bronze-DEV"
description = "Bronze layer, DEV"
capacity_id = var.capacity_id
}resource says you want this to exist. "fabric_workspace" is the kind of thing, and the provider defines it. "bronze_dev" is the name you use for it inside the code. And what goes between the braces are the arguments: what it should look like.
Here’s the most common beginner mistake: bronze_dev is not the workspace name. The name you see in the portal is display_name. To reference the workspace from another resource you use fabric_workspace.bronze_dev.id.
The HCL blocks you’ll come across
| Block | What it is |
|---|---|
resource | Something Terraform creates and manages |
variable | An input parameter: the environment, the capacity |
output | A value it returns: the ID of the workspace it created |
data | Something that already exists and you only want to look up |
locals | Calculated values you reuse: names, tags |
module | A reusable package: your template, packaged |
For a first example, resource and variable are enough. Modules come in when you start repeating the same pattern across projects.
Your first .tf for Microsoft Fabric
terraform {
required_providers {
fabric = {
source = "microsoft/fabric"
version = "~> 1.14"
}
}
}
provider "fabric" {} # authentication via environment variables
variable "capacity_id" {
type = string
}
resource "fabric_workspace" "bronze_dev" {
display_name = "Bronze-DEV"
capacity_id = var.capacity_id
}That’s four blocks: which providers you need and which version (Terraform downloads them on init), how you connect, what changes between environments and what you want to exist.
How it works inside
If you only keep one idea from this article, make it this one. Terraform always compares three things:
- Your code (
*.tf): what you want to exist. - The state (
terraform.tfstate): what Terraform knows it created and manages. - Reality (Microsoft Fabric): what’s actually there right now.
Every plan works out the differences between the three and proposes what to do so reality matches the code.
State looks a lot like the _delta_log
For people coming from data, this is the analogy that’s worked best for me:
| Delta Lake | Terraform |
|---|---|
| The parquet files are the data | The workspaces are reality |
_delta_log says which files make up the current version | terraform.tfstate says which resources it manages and their IDs |
| Without the log: loose parquet files and no table | Without the state: loose workspaces Terraform doesn’t know are its own |
The state is what lets Terraform match each resource in the code to its real identifier in Fabric. If you lose it, the code isn’t enough: Terraform won’t know those workspaces are its own and will try to create them again.
That’s why it’s worth handling with care from day one. Don’t commit it to Git, because it stores values in plain text, sometimes secrets (*.tfstate goes in .gitignore). And as soon as more than one person works on it, keep it in a remote backend with locking, an Azure storage account for example, so you all share the same state and two apply runs don’t step on each other:
terraform {
backend "azurerm" {
resource_group_name = "rg-tfstate"
storage_account_name = "sttfstate001"
container_name = "tfstate"
key = "fabric.tfstate"
}
}terraform plan is your execution plan
Would you run a 40-minute query without looking at the execution plan? With Terraform the rule’s the same: read the plan before every apply.
The symbols you need to be able to read:
| Symbol | What it means |
|---|---|
+ | Something new will be created |
~ | Updated in place, without destroying |
- | Deleted |
-/+ | Destroyed and created again: whatever was inside is lost |
This is what a real one looks like:
$ terraform plan
Terraform will perform the following actions:
# fabric_workspace.bronze_dev will be created
+ resource "fabric_workspace" "bronze_dev" {
+ display_name = "Bronze-DEV"
+ capacity_id = "f1a2b3c4-..."
+ id = (known after apply)
}
Plan: 1 to add, 0 to change, 0 to destroy.plan doesn’t change anything, so you can run it as often as you like. Always look at the last line: if something shows up under “to destroy” that you didn’t expect, stop and check. The (known after apply) values are the ones that don’t exist yet because Fabric generates them when it creates the resource.
The working cycle
| Command | What it does |
|---|---|
terraform init | Downloads the providers and sets up the backend |
terraform plan | Calculates the difference and shows it to you, touching nothing |
terraform apply | Makes the changes and updates the state |
terraform destroy | Deletes everything that configuration manages |
While you’re developing, what you’ll run most is plan and apply; you need init again when you change providers or backend. In a team, the usual setup is for the plan to show up in the pull request and for a pipeline to run the apply once it’s approved.
For data people there’s an easy way to remember it: apply behaves like a MERGE, not an INSERT. You can run it ten times and you still have the same workspaces.
What if someone touches the portal by hand?
Someone goes into the portal and removes a role “just for a moment, to test something”. Next time you run plan, Terraform checks reality, compares it with the state and shows you that change as something it’s going to put back.
That’s called drift. In the demo we caused it on purpose by removing the Viewer group from ventas-silver-prod, because it’s exactly the kind of change nobody sees without Terraform until something breaks.
The Terraform provider for Microsoft Fabric
Microsoft’s official provider is open source and releases often, so pin the version with ~> so it doesn’t change on you. Resources still in preview need preview = true in the provider block. And something worth knowing before you take it to a client: according to its own README, it isn’t covered by the Microsoft Fabric support policies; problems are reported as issues in its GitHub repo.
With it you can create structure (workspaces, roles, folders, domains and, with azurerm, the capacity), data items (lakehouse, warehouse, notebook, pipeline, semantic model, report) and platform pieces (Git integration, deployment pipelines, Spark environments, connections). The full list is in the Terraform Registry and changes with every release.
Authentication is where you lose the most time the first time round. You need a service principal in Entra ID, which is the identity Terraform shows up as. In the Fabric admin portal, the Service principals can use Fabric APIs tenant setting has to be enabled for a group that service principal belongs to; otherwise everything returns 401. To create workspaces, that group also has to be allowed to create them (Create workspaces); otherwise the first workspace fails with FeatureNotAvailable. And in Azure and Entra ID, permissions on the subscription to create the capacity and permission to create groups.
The service principal’s secrets don’t go in the .tf files: environment variables or Key Vault.
What we do and don’t do with Terraform
The question that always comes up at talks is whether this replaces deployment pipelines. It doesn’t: they cover different things.
We use Terraform for what holds the project up: workspaces and capacities, roles and groups, folders, domains and each workspace’s Git connection. The content (notebooks, semantic models, reports, pipelines) and promoting it between environments we leave to fabric-cicd, deployment pipelines or Git integration.
What we ran into while preparing the demo
A few things weren’t that clear from reading the docs:
- The capacity ID.
azurermcreates the capacity and gives you back its Azure resource ID, but a Fabric workspace expects the Fabric GUID. It cost me a failed plan; I solved it with a Fabric data source that looks the capacity up by name. I show it in the hands-on part. - A paused capacity. With the capacity paused, the Fabric provider won’t let you
plan,applyordestroy. You have to resume it and wait until it’s Active before running anything. - PowerShell 5.1. An argument like
-var-file=environments/ventas.tfvarswithout quotes gets split in two, and Terraform replies with a “Too many command line arguments” that doesn’t help at all.
And a few typical mistakes when you start: keeping the state only on your laptop, not looking at the -/+ in the plan (recreating a workspace means losing what was inside) and renaming a resource in the code without the moved block, which to Terraform means deleting one and creating another.
If you want to get started
Don’t try to automate the whole platform in one go. Install Terraform (winget install HashiCorp.Terraform), create a .tf file with a single test workspace, like the one above, and run terraform plan before applying anything. Then apply, change something in the code, look at the plan again and watch how the state gets updated. When you’re done, destroy on that test workspace (and only that one) puts everything back as it was.
A small example makes the cycle much easier to understand than copying twenty resources at once.
In the next article we see it in action: an F2 capacity, nine workspaces, 36 groups and 36 role assignments, 84 resources in a single apply, with the demo video.
Pris, my cat, has been doing declarative infrastructure for years: she declares the sofa is hers and reality converges on its own.
Useful links
Found it useful? Clap, no account needed. You can clap more than once.
Comments