In our previous Azure post, we learnt about Using Azure Sentinel for Cloud Security and Threat Detection. In this post we shall learn about Azure CycleCloud, it’s integrations, pricing and a tutorial on how to create a usage alert.
What is CycleCloud exactly?
It is the simplest way to manage HPC workloads. It is an enterprise friendly tool for orchestrating and managing High Performance Computing (HPC) environments on Azure.
Targeted at HPC administrators and users who want to deploy an HPC environment with a specific scheduler in mind which commonly used schedulers such as Slurm, PBS Pro, LSF, Grid Engine, and HT-Condor are supported out of the box.
Azure CloudCycle Features
- Orchestrate job, data and cloud workflows
- Deploy full clusters and other resources, including scheduler, compute VMs, storage, networking and cache
- Give admins full control over which users can run jobs, as well as where and at what cost
- Take advantage of built-in auto scaling and battle tested reference architectures for a wide range of HPC workloads and industries
- Use your current job scheduler and applications without modification
- Customize and optimize clusters through advanced policy and governance features, including cost controls, Active Directory integration, monitoring and reporting
Monitor, Control and Customize your Azure CloudCycle clusters
CycleCloud integrates with your Active Directory or LDAP server and provides role-based access control in helping to protect your valuable and sensitive HPC workloads. Cost alerting and controls plus performance monitoring help you get the most value from your environments.
Provision your Azure resources including virtual machines and disks to your specifications. Run Linux or Windows operating systems with your application stack installed and configured.
Define clusters with templates so you can easily create new copies of each cluster. Easily give each user or application their own dedicated cluster.
Allows to easily create and manage HPC clusters
Simplify HPC cluster configuration and easily deploy validated, enterprise ready reference architectures for common workload management across many industries. Deploy entire clusters including networking, storage, compute and the entire software stack with just a few clicks.
Automate scaling the HPC Azure CloudCycle cluster to any size
Your resource demands evolve over time and your cluster should, too. With scheduler-aware auto-scaling, you can fit your resources to your workload.
Now you can use any job scheduler or software stack
CycleCloud supports any job scheduler or software stack from proprietary in-house to open-source, third-party and commercial applications.
Enabling Hybrid HPC
With support for Avere, Microsoft HPC Pack and built-in data transfer tools, CycleCloud enables burst and hybrid HPC environments. Simply run jobs in the appropriate environment.
Azure CycleCloud vs Azure Batch
CycleCloud is the sister product to Azure Batch, which provides a Scheduler as a Service on Azure.
Also, users can provision infrastructure for HPC systems, deploy familiar HPC schedulers, and automatically scale the infrastructure to run jobs efficiently at any scale.
Users can also create different types of file systems and mount them to the compute cluster nodes to support HPC workloads.
It deploys auto-scaling plugins on top of the supported schedulers, so users do not need to implement complex auto-scaling functions and routines themselves, but rather interface only with scheduler-level configurations that they are familiar with.
Organizations who have operated HPC environments for a while typically accumulate years of expertise and in-house tooling around a specific scheduler, and re-architecting or deploying these environments on Azure can be daunting.
CycleCloud abstracts away the basic Azure building blocks such as VMs, scalesets, network interfaces, and disks. This allows an HPC administrator to focus on the familiar: an HPC cluster comprising of nodes and a configurable scheduler of choice.
With a declarative, temple based format, CycleCloud provides powerful tooling to construct complete HPC environments on Azure. Users can deploy environments that include NFS servers, parallel file systems, login hosts, license servers, and directory services.
That is essentially all the components needed in an HPC system through a single management plane.
CycleCloud integrates with other Azure services such as Azure Monitor. Below services can also be used on Azure
- Grid Engine
- HPC Pack
- PBS Pro
CycleCloud Event Grid Integration
CycleCloud can be configured to publish to an Event Grid topic by connecting it in the CycleCloud Settings page in CycleCloud.
Event Grid event subscriptions can be attached to the topic to route the events to a destination, such as a Storage Queue, where a program can consume events and process them.
CycleCloud 8.0 generates events when certain changes happen e.g. when a node is created, or a cluster is deleted.
Some events are instantaneous (e.g., deleting a cluster), and some events represent transitions e.g. creating a node which implies creating a VM. In these cases, the event is sent at the end of the transition, whether successful or not.
Azure CycleCloud Tutorial – Creating Usage Tracking
It will monitor a number of resources, including instances and volumes, and use this to estimate runtime and charges per resource. It attempts to track cluster usage.
CycleCloud allows you to set an alert per cluster that will display your usage and notify you if that usage exceeds a specified dollar amount.
To display your cluster usage, click on Create New Alert within the cluster to open the alert window. Enable the alert via the checkbox if you wish to track the spend for this particular cluster. Set a budget for the cluster, and use the dropdown to select whether the set budget is for the day or the month. Click Save to activate the alert. On the cluster page, you will see the current spend for the day or the month (depending on your selection).
Note – Usage Alerts are informational only. Going over your budget does not stop or pause the instance. The spend amount per cluster is cumulative per time period selected.
Current expenditure within budget
Current expenditure with overage
Azure CycleCloud Pricing
CycleCloud collects pricing for instances running in all cloud providers. The data is updated constantly throughout the day to provide accurate estimates and it includes below
- Boot disks and other disks (7.8.0+)
- Base price of VMs, including preview-only machine types, at standard “pay-as-you-go” rates
- Linux/Windows rates
- Reserved-VM rates
- Other software charges
- Volume-usage discounts
Assumption – All usage and cost numbers are based on approximations.
Azure CloudCycle CLI Installer File
CLI installer is distributed as part of the CycleCloud installer package.
Download Azure CloudCycle
As we saw in this post, CycleCloud is targeted at HPC administrators and other users who want to deploy an HPC environment with a specific scheduler such as Slurm, PBSPro, LSF, Grid Engine, and HT-Condor. We also saw how CycleCloud is the sister product to Azure Batch, which provides a Scheduler as a Service on Azure.