Self-hosted vs Microsoft-hosted agents for data platform CI/CD workloads.
Datanest Digital — https://datanest.dev
Choosing the right agent pool strategy directly impacts pipeline execution time,
cost, and security. This guide provides sizing recommendations for the four pipeline
types included in this product, with concrete guidance on when to use Microsoft-hosted
agents vs self-hosted agents.
| Factor | Microsoft-Hosted | Self-Hosted |
|---|---|---|
| Setup effort | None | Medium to High |
| Maintenance | None (managed by Microsoft) | OS patching, tool updates |
| Cost model | Per-minute (free tier: 1,800 min/month) | Infrastructure cost only |
| Network access | Public internet only | Private VNET, on-premises |
| Startup latency | 30-90 seconds | Near-instant (if pre-warmed) |
| Disk space | ~10 GB free | Custom (100+ GB possible) |
| Pre-installed tools | Standard set | Fully customizable |
| Parallel jobs | 1 free, then paid | Limited by agent count |
| Stateful caching | No (ephemeral VMs) | Yes (persistent disk) |
| Private endpoint access | No | Yes |
Recommended: Self-hosted (for production), Microsoft-hosted (for dev)
| Reason |
|---|
| Terraform state storage may be behind a private endpoint |
| State files can be large; persistent disk caching avoids re-download |
| Provider plugins (~500 MB) benefit from disk caching between runs |
| Production infrastructure changes need network access to private VNETs |
Agent Sizing:
Recommended: Microsoft-hosted (all environments)
| Reason |
|---|
| Databricks CLI communicates over HTTPS to workspace public endpoints |
| No large local dependencies; CLI is a small binary |
| Python linting and unit tests run quickly on standard agents |
| No persistent state needed between runs |
If using private-link Databricks workspaces: Switch to self-hosted agents
with VNET connectivity to the workspace's private endpoint.
Agent Sizing (if self-hosted):
Recommended: Microsoft-hosted (all environments)
| Reason |
|---|
| ARM template deployments use Azure Resource Manager public APIs |
| No large binaries or build artifacts |
| Pipeline is lightweight and short-running (typically 2-5 minutes) |
Agent Sizing (if self-hosted):
Recommended: Self-hosted (for staging and production)
| Reason |
|---|
| SQL servers are typically behind private endpoints or firewall rules |
| DACPAC builds require .NET SDK which benefits from NuGet package caching |
| sqlpackage operations against large databases can be memory-intensive |
| Schema drift reports need direct database connectivity |
Agent Sizing:
| Option | Pros | Cons |
|---|---|---|
| Azure VM Scale Sets (VMSS) | Auto-scaling, managed lifecycle | Higher complexity |
| Azure Container Instances | Fast startup, cost-efficient | No persistent disk |
| Azure Kubernetes Service | Scalable, reusable clusters | Operational overhead |
| Single Azure VM | Simple, cheap | No auto-scaling |
Azure VM Scale Sets provide the best balance of auto-scaling, cost control, and
maintenance simplicity for self-hosted agents.
# Create a VMSS for pipeline agents
az vmss create \
--name "vmss-azdo-agents" \
--resource-group "rg-azdo-agents" \
--image "Ubuntu2204" \
--vm-sku "Standard_D2s_v5" \
--instance-count 0 \
--upgrade-policy-mode manual \
--vnet-name "vnet-data-platform" \
--subnet "snet-azdo-agents" \
--admin-username "azureagent" \
--generate-ssh-keysRegister as an Azure DevOps agent pool:
1. Go to Organization Settings > Agent pools > Add pool.
2. Select Azure virtual machine scale set.
3. Select the VMSS resource and configure:
| Setting | Recommended Value |
|---|---|
| Maximum number of VMs | 4 |
| Number of agents to keep on standby | 1 |
| Delay in minutes before deleting excess idle agents | 30 |
| Grace period (minutes) | 15 |
Install these tools on the self-hosted agent image:
| Tool | Version | Required By |
|---|---|---|
| Azure CLI | Latest | All pipelines |
| Terraform | 1.7.x | Terraform pipeline |
| Databricks CLI | 0.232.x | DABs pipeline |
| .NET SDK | 8.x | SQL migration pipeline |
| sqlpackage | Latest | SQL migration pipeline |
| Python | 3.11.x | DABs pipeline, linting |
| Git | Latest | All pipelines |
| jq | Latest | Various script steps |
| ruff | 0.4.x | Python linting |
| Docker (optional) | Latest | Container-based builds |
Not recommended for production deployments due to eviction risk.
| Scenario | Microsoft-Hosted | Self-Hosted (VMSS) |
|---|---|---|
| 50 pipeline runs/month, 10 min avg | Free tier | ~$15 (D2s_v5 spot) |
| 200 runs/month, 10 min avg | ~$40 (1 extra parallel) | ~$45 (D2s_v5) |
| 500 runs/month, 15 min avg | ~$120 (3 extra parallel) | ~$90 (D2s_v5) |
1. Network isolation: Place self-hosted agents in a dedicated subnet with NSG rules
that only allow outbound access to required endpoints.
2. Ephemeral agents: Configure VMSS agents to reimage after every job to prevent
credential leakage between pipeline runs.
3. Managed identity: Assign a system-assigned managed identity to the VMSS and use
it for Azure authentication instead of service principal secrets.
4. Agent updates: Enable automatic agent software updates in the pool configuration.
5. Firewall rules: Microsoft-hosted agents use dynamic IP ranges. If your resources
require IP allowlisting, self-hosted agents with a static outbound IP (via NAT Gateway)
are the only reliable option.
Copyright 2024-2026 Datanest Digital. All rights reserved.
Get the full Azure DevOps for Data Pipelines and unlock everything.
Get the complete guide with every chapter unlocked, including code samples, diagrams, and best practices.
Access all interactive tools with complete data, all workload profiles, and the full scenario library.
Downloadable source code, configuration files, and working examples from every chapter.
Free updates for life. Every new chapter, tool, and improvement included.