cluster
Defines one or more clusters for different job types or workloads.
Each cluster can have its own individual configuration.
The format is [cluster <clustername>].
key_name
Name of an existing EC2 KeyPair to enable SSH access to the instances.
template_url
Defines the path to the CloudFormation template used to create the cluster.
Updates use the template the stack was created with.
Defaults to
https://s3.amazonaws.com/<aws_region_name>-aws-parallelcluster/templates/aws-parallelcluster-<version>.cfn.json.
template_url = https://s3.amazonaws.com/us-east-1-aws-parallelcluster/templates/aws-parallelcluster.cfn.json
compute_instance_type
Defines the EC2 instance type used for the cluster compute nodes.
If the scheduler is awsbatch, please refer to the Compute Environments creation in the
AWS Batch UI for the list of supported instance types.
Defaults to t2.micro, optimal when scheduler is awsbatch
compute_instance_type = t2.micro
master_instance_type
Defines the EC2 instance type used for the master node.
Defaults to t2.micro.
master_instance_type = t2.micro
initial_queue_size
Set the initial number of EC2 instances to launch as compute nodes in the cluster.
This setting is applicable only for traditional schedulers (sge, slurm, and torque).
If the scheduler is awsbatch, use min_vcpus.
Defaults to 2.
max_queue_size
Set the maximum number of EC2 instances that can be launched in the cluster.
This setting is applicable only for traditional schedulers (sge, slurm, and torque).
If the scheduler is awsbatch, use max_vcpus.
Defaults to 10.
maintain_initial_size
Boolean flag to maintain initial size of the Auto Scaling group for traditional schedulers.
If the scheduler is awsbatch, use desired_vcpus.
If set to true, the Auto Scaling group will never have fewer members than the value
of initial_queue_size. The cluster can still scale up to the value of max_queue_size.
If set to false, the Auto Scaling group can scale down to 0 members to prevent resources
from sitting idle when they are not needed.
Defaults to false.
maintain_initial_size = false
min_vcpus
If the scheduler is awsbatch, the compute environment will never have fewer than min_vcpus.
Defaults to 0.
desired_vcpus
If the scheduler is awsbatch, the compute environment will initially have desired_vcpus.
Defaults to 4.
max_vcpus
If the scheduler is awsbatch, the compute environment will at most have max_vcpus.
Defaults to 20.
scheduler
Defines the cluster scheduler.
Valid options are sge, torque, slurm, or awsbatch.
If the scheduler is awsbatch, please take a look at the networking setup.
Defaults to sge.
cluster_type
Defines the type of cluster to launch.
Valid options are ondemand or spot.
Defaults to ondemand.
spot_price
If cluster_type is set to spot, you can optionally set the maximum spot price for the
ComputeFleet on traditional schedulers. If you do not specify a value, you are charged the
Spot price, capped at the On-Demand price.
If the scheduler is awsbatch, use spot_bid_percentage.
See the Spot Bid Advisor for assistance finding a bid price that meets your needs.
spot_bid_percentage
If awsbatch is the scheduler, this optional parameter is the on-demand bid percentage.
If unspecified, the current spot market price will be selected, capped at the on-demand price.
custom_ami
ID of a Custom AMI to use instead of the default published AMIs.
s3_read_resource
Specify an S3 resource to which AWS ParallelCluster nodes will be granted read-only access.
For example, ‘arn:aws:s3:::my_corporate_bucket/*’ would provide read-only access to all
objects in the my_corporate_bucket bucket.
See working with S3 for details on format.
Defaults to NONE.
s3_read_write_resource
Specify an S3 resource to which AWS ParallelCluster nodes will be granted read-write access.
For example, ‘arn:aws:s3:::my_corporate_bucket/Development/*’ would provide read-write
access to all objects in the Development folder of the my_corporate_bucket bucket.
See working with S3 for details on format.
Defaults to NONE.
s3_read_write_resource = NONE
pre_install
URL to a preinstall script that is executed before any of the boot_as_* scripts are run.
When using awsbatch as the scheduler, the preinstall script is only executed on the master node.
The parameter format can be specified as “http://hostname/path/to/script.sh” or “s3://bucketname/path/to/script.sh”.
Defaults to NONE.
pre_install_args
Quoted list of arguments to be passed to the preinstall script.
Defaults to NONE.
post_install
URL to a postinstall script that is executed after all of the boot_as_* scripts are run.
When using awsbatch as the scheduler, the postinstall script is only executed on the master node.
Can be specified in “http://hostname/path/to/script.sh” or “s3://bucketname/path/to/script.sh” format.
Defaults to NONE.
post_install_args
Arguments to be passed to the postinstall script.
Defaults to NONE.
placement_group
Defines the cluster placement group.
Valid options are NONE, DYNAMIC or an existing EC2 placement group name.
When DYNAMIC is set, a unique placement group will be created and deleted as part
of the cluster stack.
This parameter does not apply to awsbatch.
More information on placement groups can be found here
Defaults to NONE.
placement
Defines the cluster placement group logic.
This enables the whole cluster or only the compute instances to use the placement group.
Valid options are cluster or compute.
This parameter does not apply to awsbatch.
Defaults to compute.
ephemeral_dir
If instance store volumes exist, define the path where they will be mounted.
Defaults to /scratch.
shared_dir
Defines the path where the shared EBS volume will be mounted.
Do not use this option with multiple EBS volumes. Provide shared_dir under each EBS section instead.
See EBS Section for details on working with multiple EBS volumes.
Defaults to /shared.
The example below mounts the shared EBS volume at /myshared.
encrypted_ephemeral
Encrypt the ephemeral instance store volumes with non-recoverable in-memory keys
using LUKS (Linux Unified Key Setup).
Please visit https://guardianproject.info/code/luks/ for more information.
Defaults to false.
encrypted_ephemeral = false
master_root_volume_size
MasterServer root volume size in GB. The AMI must support growroot.
Defaults to 17, min value 17.
master_root_volume_size = 17
compute_root_volume_size
ComputeFleet root volume size in GB. The AMI must support growroot.
Defaults to 17, min value 17.
compute_root_volume_size = 17
base_os
OS type used in the cluster.
Available options are: alinux, centos6, centos7, ubuntu1404 and ubuntu1604.
Supported operating systems by region are listed in the table below. Please note
that commercial entails all supported regions including us-east-1, us-west-2, etc.:
============== ====== ============ ============ ============= ============
region alinux centos6 centos7 ubuntu1404 ubuntu1604
============== ====== ============ ============ ============= ============
commercial True True True True True
govcloud True False False True True
china True False False True True
============== ====== ============ ============ ============= ============
Note: The base_os determines the username used to log into the cluster.
- CentOS 6 & 7:
centos
- Ubuntu 14.04 LTS & 16.04 LTS:
ubuntu
- Amazon Linux:
ec2-user
Defaults to alinux.
ec2_iam_role
Defines the name of an existing EC2 IAM Role that will be attached to all instances in
the cluster. Note that the given name of a role and its Amazon Resource Name (ARN) are
different, and the latter may not be used as an argument to ec2_iam_role.
Defaults to NONE.
additional_cfn_template
Defines an additional CloudFormation template to launch along with the cluster. This
allows for the creation of resources that exist outside of the cluster but are part
of the cluster’s life cycle.
This value must be a HTTP URL to a public template with all parameters provided.
Defaults to NONE.
additional_cfn_template = NONE
vpc_settings
Settings section for the VPC where the cluster will be deployed.
See VPC Section.
ebs_settings
Settings section related to the EBS volume mounted on the master instance. When using
multiple EBS volumes, enter these parameters as a comma separated list.
Up to five (5) additional EBS volumes are supported.
See EBS Section.
ebs_settings = custom1, custom2, ...
scaling_settings
Settings section relating to autoscaling configuration.
See Scaling Section.
scaling_settings = custom
efs_settings
Settings section relating to EFS filesystem.
See EFS Section.
raid_settings
Settings section relating to EBS volume RAID configuration.
See RAID Section.
fsx_settings
Settings section relating to FSx Lustre configuration.
See FSx Section.
EFS
Defines configuration settings for the EFS mounted on the master and compute instances.
[efs customfs]
shared_dir = efs
encrypted = false
performance_mode = generalPurpose
shared_dir
Defines the EFS mount point on the master and compute nodes.
This parameter is REQUIRED! The EFS section will only be used if shared_dir is specified.
The example below will mount at /efs.
Do not use NONE or /NONE as the shared directory.:
encrypted
Defines if the file system will be encrypted.
Defaults to false.
throughput_mode
Defines the Throughput Mode of the file system.
Valid options are bursting and provisioned.:
throughput_mode = provisioned
provisioned_throughput
Defines the provisioned throughput measured in MiB/s.
This parameter requires setting throughput_mode to provisioned.
The limit on throughput is 1024 MiB/s. Please contact AWS Support to request a limit increase.
Valid Range: Min of 0.0.:
provisioned_throughput = 1024
efs_fs_id
Defines the EFS file system ID for an existing file system.
Specifying this option will void all other EFS options except for shared_dir.
config_sanity will only support file systems without a mount target in the stack’s
availability zone or file systems that have an existing mount target in the stack’s
availability zone with inbound and outbound NFS traffic allowed from 0.0.0.0/0.
The sanity check for validating efs_fs_id requires the IAM role to have the following permissions:
efs:DescribeMountTargets
efs:DescribeMountTargetSecurityGroups
ec2:DescribeSubnets
ec2:DescribeSecurityGroups
Please add these permissions to your IAM role or set sanity_check = false to avoid errors.
CAUTION: Having mount target with inbound and outbound NFS traffic allowed from 0.0.0.0/0
will expose the file system to NFS mounting request from anywhere in the mount target’s
availability zone. AWS recommends not creating a mount target in the stack’s availability
zone and letting us handle this step. If you must have a mount target in the stack’s
availability zone, please consider using a custom security group by providing a vpc_security_group_id
option under the vpc section, adding that security group to the mount target, and turning
off config sanity to create the cluster.
Defaults to NONE.:
RAID
Defines configuration settings for a RAID array built from a number of identical
EBS volumes.
The RAID drive is mounted on the master node and exported to compute nodes via NFS.
[raid rs]
shared_dir = raid
raid_type = 1
num_of_raid_volumes = 2
encrypted = true
shared_dir
Defines the mount point for the RAID array on the master and compute nodes.
The RAID drive will only be created if this parameter is specified.
The example below will mount the array at /raid.
Do not use NONE or /NONE as the shared directory.:
raid_type
Defines the RAID type for the RAID array.
Valid options are RAID 0 or RAID 1.
For more information on RAID types, see: RAID info
The RAID drive will only be created if this parameter is specified.
The example below will create a RAID 0 array:
num_of_raid_volumes
Defines the number of EBS volumes to assemble the RAID array from.
Minimum number of volumes = 2.
Maximum number of volumes = 5.
Defaults to 2.
volume_type
Defines the type of volume to build.
See: Volume type for more detail.
Defaults to gp2.
volume_size
Defines the size of volume to be created.
Defaults to 20GB.
volume_iops
Defines the number of IOPS for io1 type volumes.
encrypted
Determines if the file system will be encrypted.
Defaults to false.
ebs_kms_key_id
Use a custom KMS Key for encryption.
This must be used in conjunction with encrypted = true and must have a custom ec2_iam_role.
See Disk Encryption with a Custom KMS Key.
ebs_kms_key_id = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
FSx
Configuration for an attached FSx Lustre file system. See FSx CreateFileSystem for more information.
FSx Lustre is supported when base_os = centos7 | alinux.
When using an Amazon Linux custom_ami, the kernel must be >= 4.14.104-78.84.amzn1.x86_64.
See Installing the Lustre Client
for instructions.
Note FSx is not currently supported when using awsbatch as a scheduler.
If using an existing file system, it must be associated to a security group that allows inbound and outbound
TCP traffic from 0.0.0.0/0 through port 988. This is done by automatically when not using
vpc_security_group_id.
Use an existing FSx file system by specifying fsx_fs_id.
[fsx fs]
shared_dir = /fsx
fsx_fs_id = fs-073c3803dca3e28a6
Or create and configure a new file system, with the following parameters
[fsx fs]
shared_dir = /fsx
storage_capacity = 3600
import_path = s3://bucket
imported_file_chunk_size = 1024
export_path = s3://bucket/folder
weekly_maintenance_start_time = 1:00:00
shared_dir
Required Defines the mount point for the Lustre File system on the master and compute nodes.
The example below will mount the filesystem at /fsx.
Do not use NONE or /NONE as the shared directory.:
fsx_fs_id
Optional Attach an existing FSx File System.
If this option is specified, all following FSx parameters, such as storage_capacity are ignored.
fsx_fs_id = fs-073c3803dca3e28a6
storage_capacity
Optional The storage capacity of the file system in GiB.
The storage capacity has a minimum of 3,600 GiB and is provisioned in increments of 3,600 GiB.
Defaults to 3,600 GiB.
import_path
Optional S3 Bucket to load data from into the file system. Also serves as the export bucket. See export_path.
Import occurs on cluster creation, see Importing Data from your Amazon S3 Bucket
If not provided, file system will be empty.
import_path = s3://bucket
imported_file_chunk_size
Optional For files imported from a data repository (using import_path), this value determines the stripe count
and maximum amount of data per file (in MiB) stored on a single physical disk. The maximum number of disks that a single
file can be striped across is limited by the total number of disks that make up the file system.
The chunk size default is 1,024 MiB (1 GiB) and can go as high as 512,000 MiB (500 GiB).
Amazon S3 objects have a maximum size of 5 TB.
Valid only when using import_path.
imported_file_chunk_size = 1024
export_path
Optional The S3 path where the root of your file system is exported. The path must be in the same S3 bucket as
the import_path parameter.
Defaults to s3://import-bucket/FSxLustre[creation-timestamp] where import-bucket is the bucket provided in
import_path parameter.
Valid only when using import_path.
export_path = s3://bucket/folder
weekly_maintenance_start_time
*Optional Preferred time to perform weekly maintenance, in UTC time zone.
Format is [day of week]:[hour of day]:[minute of hour]. For example, Monday at Midnight is:
weekly_maintenance_start_time = 1:00:00
