Run high-availability long-running clusters with Amazon EMR instance fleets
AWS now supports high availability Amazon EMR on EC2 clusters with instance fleet configuration. With high availability instance fleet clusters, you now get the enhanced resiliency and fault tolerance of high availability architecture, along with the improved flexibility and intelligence in Amazon Elastic Compute Cloud (Amazon EC2) instance selection of instance fleets. Amazon EMR is a cloud big data platform for petabyte-scale data processing, interactive analysis, streaming, and machine learning (ML) using open source frameworks such as Apache Spark, Presto and Trino, and Apache Flink. Customers love the scalability and flexibility that Amazon EMR on EC2 offers. However, like most distributed systems running mission-critical workloads, high availability is a core requirement, especially for those with long-running workloads.
In this post, we demonstrate how to launch a high availability instance fleet cluster using the newly redesigned Amazon EMR console, as well as using an AWS CloudFormation template. We also go over the basic concepts of Hadoop high availability, EMR instance fleets, the benefits and trade-offs of high availability, and best practices for running resilient EMR clusters.
High availability in Hadoop
High availability (HA) provides continuous uptime and fault tolerance for a Hadoop cluster. The core components of Hadoop, like Hadoop Distributed File System (HDFS) NameNode and YARN ResourceManager, are single points of failure in clusters with a single primary node. In the event that any of them crash, the entire cluster goes down. High Availability removes this single point of failure by introducing redundant standby nodes that can quickly take over if the primary node fails.
In a high availability EMR cluster, one node serves as the active NameNode that handles client operations, and others act as standby NameNodes. The standby NameNodes constantly synchronize their state with the active one, enabling seamless failover to maintain service availability. To learn more, see Supported applications in an Amazon EMR Cluster with multiple primary nodes.
Key instance fleet differentiations
Amazon EMR recommends using the instance fleet configuration option for provisioning EC2 instances in EMR clusters because it offers a flexible and robust approach to cluster provisioning. Some key advantages include:
- Flexible instance provisioning – Instance fleets provide a powerful and simple way to specify up to five EC2 instance types on the Amazon EMR console, or up to 30 when using the AWS Command Line Interface (AWS CLI) or API with an allocation strategy. This enhanced diversity helps optimize for cost and performance while increasing the likelihood of fulfilling capacity requirements.
- Target capacity management – You can specify target capacities for On-Demand and Spot Instances for each fleet. Amazon EMR automatically manages the mix of instances to meet these targets, reducing operational overhead.
- Improved availability – By spanning multiple instance types and purchasing options such as On-Demand and Spot, instance fleets are more resilient to capacity fluctuations in specific EC2 instance pools.
- Enhanced Spot Instance handling – Instance fleets offer superior management of Spot Instances, including the ability to set timeouts and specify actions if Spot capacity can’t be provisioned.
- Reliable cluster launches – You can configure your instance fleet to select multiple subnets for different Availability Zones, allowing Amazon EMR to find the best combination of instances and purchasing options across these zones to launch your cluster in. Amazon EMR will identify the best Availability Zone based on your configuration and available EC2 capacity and launch the cluster.
Prerequisites
Before you launch the high availability EMR instance fleet clusters, make sure you have the following:
- Latest Amazon EMR release – We recommend that you use the latest Amazon EMR release to benefit from the highest level of resiliency and stability for your high availability clusters. High availability for instance fleets is supported with Amazon EMR releases 5.36.1, 6.8.1, 6.9.1, 6.10.1, 6.11.1, 6.12.0, and later.
- Supported applications – High availability for instance fleets is supported for applications such as Apache Spark, Presto, Trino, and Apache Flink. Refer to Supported applications in an Amazon EMR Cluster with multiple primary nodes for the complete list of supported applications and their failover processes.
Launch a high availability instance fleet cluster using the Amazon EMR console
Complete the following steps on the Amazon EMR console to configure and launch a high availability EMR cluster with instance fleets:
- On the Amazon EMR console, create a new cluster.
- For Name, enter a name.
- For Amazon EMR release, choose the Amazon EMR release that supports high availability clusters with instance fleets. The setting will default to the latest available Amazon EMR release.
- Under Cluster configuration, choose the desired instance types for the primary fleet. (You can select up to five when using the Amazon EMR console.)
- Select Use high availability to launch the cluster with three primary nodes.
- Choose the instance types and target On-Demand and Spot size for the core and task fleet according to your requirements.
- Under Allocation strategy, select Apply allocation strategy.
- 1 We recommend that you select Price-capacity optimized for your allocation strategy for your cluster for faster cluster provisioning, more accurate Spot Instance allocation, and fewer Spot Instance interruptions.
- Under Networking, you can choose multiple subnets for different Availability Zones. This allows Amazon EMR to look across those subnets and launch the cluster in an Availability Zone that best suits your instance and purchasing option requirements.
- Review your cluster configuration and choose Create cluster.
Amazon EMR will launch your cluster in a few minutes. You can view the cluster details on the Amazon EMR console.
Launch a high availability cluster with AWS CloudFormation
To launch a high availability cluster using AWS CloudFormation, complete the following steps:
- Create a CloudFormation template with EMR resource type
AWS::EMR::Clusterand JobFlowInstancesConfig property typesMasterInstanceFleet,CoreInstanceFleetand (optional)TaskInstanceFleets. To launch a high availability cluster, configure TargetOnDemandCapacity=3, TargetSpotCapacity=0 for the primary instance fleet and weightedCapacity=1 for each instance type configured for the fleet. See the following code:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"cluster": {
"Type": "AWS::EMR::Cluster",
"Properties": {
"Instances": {
"Ec2SubnetIds": [
"subnet-003c889b8379f42d1",
"subnet-0382aadd4de4f5da9",
"subnet-078fbbb77c92ab099"
],
"MasterInstanceFleet": {
"Name": "HAPrimaryFleet",
"TargetOnDemandCapacity": 3,
"TargetSpotCapacity": 0,
"InstanceTypeConfigs": [
{
"InstanceType": "m5.xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "m5.2xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "m5.4xlarge",
"WeightedCapacity": 1
}
]
},
"CoreInstanceFleet": {
"Name": "cfnCore",
"InstanceTypeConfigs": [
{
"InstanceType": "m5.xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "m5.2xlarge",
"WeightedCapacity": 2
},
{
"InstanceType": "m5.4xlarge",
"WeightedCapacity": 4
}
],
"LaunchSpecifications": {
"SpotSpecification": {
"TimeoutAction": "SWITCH_TO_ON_DEMAND",
"TimeoutDurationMinutes": 20,
"AllocationStrategy": "PRICE_CAPACITY_OPTIMIZED"
}
},
"TargetOnDemandCapacity": "4",
"TargetSpotCapacity": 0
},
"TaskInstanceFleets": [
{
"Name": "cfnTask",
"InstanceTypeConfigs": [
{
"InstanceType": "m5.xlarge",
"WeightedCapacity": 1
},
{
"InstanceType": "m5.2xlarge",
"WeightedCapacity": 2
},
{
"InstanceType": "m5.4xlarge",
"WeightedCapacity": 4
}
],
"LaunchSpecifications": {
"SpotSpecification": {
"TimeoutAction": "SWITCH_TO_ON_DEMAND",
"TimeoutDurationMinutes": 20,
"AllocationStrategy": "PRICE_CAPACITY_OPTIMIZED"
}
},
"TargetOnDemandCapacity": "0",
"TargetSpotCapacity": 4
}
]
},
"Name": "TestHACluster",
"ServiceRole": "EMR_DefaultRole",
"JobFlowRole": "EMR_EC2_DefaultRole",
"ReleaseLabel": "emr-6.15.0",
"PlacementGroupConfigs": [
{
"InstanceRole": "MASTER",
"PlacementStrategy": "SPREAD"
}
]
}
}
}
}
Make sure to use an Amazon EMR release that supports high availability clusters with instance fleets.
- Create a CloudFormation stack with the preceding template:
aws cloudformation create-stack --stack-name HAInstanceFleetCluster --template-body file://cfn-template.json --region us-east-1- Retrieve the cluster ID from the list-clusters response to use in the following steps. You can further filter this list based on filters like cluster status, creation date, and time.
aws emr list-clusters --query "Clusters[?Name=='<YourClusterName>']"- Run the following describe-cluster command:
aws emr describe-cluster --cluster-id j-XXXXXXXXXXX --region us-east-1If the high availability cluster was launched successfully, the describe-cluster response will return the state of the primary fleet as RUNNING and provisionedOnDemandCapacity as 3. By this point, all three primary nodes have been started successfully.
Primary node failover with High Availability clusters
To fetch information on all EC2 instances for an instance fleet, use the list-instances command:
aws emr list-instances --cluster-id j-XXXXXXXXXXX --instance-fleet-type MASTER --region us-east-1
For high availability clusters, it will return three instances in RUNNING state for the primary fleet and other attributes like public and private DNS names.
The following screenshot shows the instance fleet status on the Amazon EMR console.
Let’s examine two cases for primary node failover.
Case 1: One of the three primary instances is accidentally stopped
When an EC2 instance is accidentally stopped by a user, Amazon EMR detects this and performs a failover for the stopped primary node. Amazon EMR also attempts to launch a new primary node with the same private IP and DNS name to recover back the quorum. During this failover, the cluster remains fully operational, providing true resiliency to single primary node failures.
The following screenshots illustrate the instance fleet details.
This automatic recovery for primary nodes is also reflected in the MultiMasterInstanceGroupNodesRunning or MultiMasterInstanceGroupNodesRunningPercentage Amazon CloudWatch metric emitted by Amazon EMR for your cluster. The following screenshot shows an example of these metrics.
Case 2: One of the three primary instances becomes unhealthy
If Amazon EMR continuously receives failures when trying to connect to a primary instance, it is deemed as unhealthy and Amazon EMR will attempt to replace it. Similar to case 1, Amazon EMR will perform a failover for the stopped primary node and also attempt to launch a new primary node with the same private IP and DNS name to recover the quorum.
If you list the instances for the primary fleet, the response will include information for the EC2 instance that was stopped by the user and the new primary instance that replaced it with the same private IP and DNS name.
The following screenshot shows an example of the CloudWatch metrics.
An instance can have connection failures for multiple reasons, including but not limited to disk space unavailable on the instance, critical cluster daemons like instance controller shut down with errors, high CPU utilization, and more. Amazon EMR is continuously improving its health monitoring criteria to better identify unhealthy nodes on an EMR cluster.
Considerations and best practices
The following are some of the key considerations and best practices for using EMR instance fleets to launch a high availability cluster with multiple primary nodes:
- Use the latest EMR release – With the latest EMR releases, you get the highest level of resiliency and stability for your high availability EMR clusters with multiple primary nodes.
- Configure subnets for high availability – Amazon EMR can’t replace a failed primary node if the subnet is oversubscribed (there aren’t any available private IP addresses in the subnet). This results in a cluster failure as soon as the second primary node fails. Limited availability of IP addresses in a subnet can also result in cluster launch or scaling failures. To avoid such scenarios, we recommend that you dedicate an entire subnet to an EMR cluster.
- Configure core nodes for enhanced data availability – To minimize the risk of local HDFS data loss on your production clusters, we recommend that you set the
dfs.replicationparameter to 3 and launch at least four core nodes. Settingdfs.replicationto 1 on clusters with fewer than four core nodes can lead to data loss if a single core node goes down. For clusters with three or fewer core nodes, setdfs.replicationparameter to at least 2 to achieve sufficient HDFS data replication. For more information, see HDFS configuration. - Use an allocation strategy – We recommend enabling an allocation strategy option for your instance fleet cluster to provide faster cluster provisioning, more accurate Spot Instance allocation, and fewer Spot Instance interruptions.
- Set alarms for monitoring primary nodes – You should monitor the health and status of primary nodes of your long-running clusters to maintain smooth operations. Configure alarms using CloudWatch metrics such as
MultiMasterInstanceGroupNodesRunning,MultiMasterInstanceGroupNodesRunningPercentage, orMultiMasterInstanceGroupNodesRequested. - Integrate with EC2 placement groups – You can also choose to protect primary instances against hardware failures by using a placement group strategy for your primary fleet. This will spread the three primary instances across separate underlying hardware to avoid loss of multiple primary nodes at the same time in the event of a hardware failure. See Amazon EMR integration with EC2 placement groups for more details.
When setting up a high availability instance fleet cluster with Amazon EMR on EC2, it’s important to understand that all EMR nodes, including the three primary nodes, are launched within a single Availability Zone. Although this configuration maintains high availability within that Availability Zone, it also means that the entire cluster can’t tolerate an Availability Zone outage. To mitigate the risk of cluster failures due to Spot Instance reclamation, Amazon EMR launches the primary nodes using On-Demand instances, providing an additional layer of reliability for these critical components of the cluster.
Conclusion
This post demonstrated how you can use high availability with EMR on EC2 instance fleets to enhance the resiliency and reliability of your big data workloads. By using instance fleets with multiple primary nodes, EMR clusters can withstand failures and maintain uninterrupted operations, while providing enhanced instance diversity and better Spot capacity management within a single Availability Zone. You can quickly set up these high availability clusters using the Amazon EMR console or AWS CloudFormation, and monitor their health using CloudWatch metrics.
To learn more about the supported applications and their failover process, see Supported applications in an Amazon EMR Cluster with multiple primary nodes. To get started with this feature and launch a high availability EMR on EC2 cluster, refer to Plan and configure primary nodes.
About the Authors
Garima Arora is a Software Development Engineer for Amazon EMR at Amazon Web Services. She specializes in capacity optimization and helps build services that allow customers to run big data applications and petabyte-scale data analytics faster. When not hard at work, she enjoys reading fiction novels and watching anime.
Ravi Kumar is a Senior Product Manager Technical-ES (PMT) at Amazon Web Services, specialized in building exabyte-scale data infrastructure and analytics platforms. With a passion for building innovative tools, he helps customers unlock valuable insights from their structured and unstructured data. Ravi’s expertise lies in creating robust data foundations using open-source technologies and advanced cloud computing, that powers advanced artificial intelligence and machine learning use cases. A recognized thought leader in the field, he advances the data and AI ecosystem through pioneering solutions and collaborative industry initiatives. As a strong advocate for customer-centric solutions, Ravi constantly seeks ways to simplify complex data challenges and enhance user experiences. Outside of work, Ravi is an avid technology enthusiast who enjoys exploring emerging trends in data science, cloud computing, and machine learning.
Tarun Chanana is a Software Development Manager for Amazon EMR at Amazon Web Services.
Post Comment