Hadoop Cluster Tutorials

 

Hands-on lab on Hadoop Cluster

What is a Hadoop Cluster?

A Hadoop cluster is a collection of computers, known as nodes, that are networked together to perform parallel computations on big data sets. The Name node is the master node of the Hadoop Distributed File System (HDFS). It maintains the meta data of the files in the RAM for quick access. An actual Hadoop Cluster setup involves extensives resources which are not within the scope of this lab. In this lab, you will use dockerized hadoop to create a Hadoop Cluster which will have:

  1. Namenode
  2. Datanode
  3. Node Manager
  4. Resource manager
  5. Hadoop history server

Objectives

  • Run a dockerized Cluster Hadoop instance
  • Create a file in the HDFS and view it on the GUI

Set up Cluster Nodes Dockerized Hadoop

  1. Start online lab - https://labs.cognitiveclass.ai/login/lti
  2. Start a new terminal
  1. Clone the repository to your theia environment.
git clone https://github.com/ibm-developer-skills-network/ooxwv-docker_hadoop.git

  1. Navigate to the docker-hadoop directory to build it.
cd ooxwv-docker_hadoop

  1. Compose the docker application.
docker-compose up -d

Compose is a tool for defining and running multi-container Docker applications. It uses the YAML file to configure the serives and enables us to create and start all the services from just one configurtation file.

You will see that all the five containers are created and started.

  1. Run the namenode as a mounted drive on bash.
docker exec -it namenode /bin/bash

  1. You will observe that the prompt changes as shown below.

Explore the hadoop environment

As you have learnt in the videos and reading thus far in the course, a Hadoop environment is configured by editing a set of configuration files:

  • hadoop-env.sh Serves as a master file to configure YARN, HDFS, MapReduce, and Hadoop-related project settings.

  • core-site.xml Defines HDFS and Hadoop core properties

  • hdfs-site.xml Governs the location for storing node metadata, fsimage file and log file.

  • mapred-site-xml Lists the parameters for MapReduce configuration.

  • yarn-site.xml Defines settings relevant to YARN. It contains configurations for the Node Manager, Resource Manager, Containers, and Application Master.

For the docker image, these xml files have been configured already. You can see these in the directory /opt/hadoop-3.2.1/etc/hadoop/ by running

ls /opt/hadoop-3.2.1/etc/hadoop/*.xml

Create a file in the HDFS

  1. In the HDFS, create a directory structure named user/root/input.
hdfs dfs -mkdir -p /user/root/input

  1. Copy all the hadoop configuration xml files into the input directory.
hdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml /user/root/input

  1. Create a data.txt file in the current directory.
curl https://raw.githubusercontent.com/ibm-developer-skills-network/ooxwv-docker_hadoop/master/SampleMapReduce.txt --output data.txt

  1. Copy the data.txt file into /user/root.
hdfs dfs -put data.txt /user/root/

  1. Check if the file has been copied into the HDFS by viewing its content.
hdfs dfs -cat /user/root/data.txt

View the HDFS

  1. Click on the Skills Network button on the left, it will open the "Skills Network Toolbox". Then click the Other then Launch Application. From there you should be able to enter the port number as 9870 and launch.
  1. This will open up the Graphical User Interface (GUI) of the Hadoop node. Click on Utilities -> Broswe the file system to browse the files.
  1. View the files in the directories that you have just created by clicking on user then root.

  1. Notice that the block size is 128 MB though the file size is actually much smaller. This is because the default block size used by HDFS is 128 MB.

  2. You can click on the file to check the file into. It gives you information about the file in terms of number of bytes, block id etc.,

Congratulations! You have:

  • Deployed Hadoop using Docker
  • Created data in HDFS and viewed it on the GUI

Comments

Post a Comment

Popular posts from this blog

Terraform Commands - Hands-on Learning

Installing Terraform What is Terraform and Why do we need it? Terraform Commands:- Here, we begin working with Terraform commands. We will start by creating a very simple Terraform file that will pull down the an image from Docker Hub. List the Terraform commands: terraform Common commands: apply: Builds or changes infrastructure console: Interactive console for Terraform interpolations destroy: Destroys Terraform-managed infrastructure fmt: Rewrites configuration files to canonical format get: Downloads and installs modules for the configuration graph: Creates a visual graph of Terraform resources import: Imports existing infrastructure into Terraform init: Initializes a new or existing Terraform configuration output: Reads an output from a state file plan: Generates and shows an execution plan providers: Prints a tree of the providers used in the configuration push: Uploads this Terraform module to Terraform Enterprise to run refresh: Updates local state file against real resources s...
Read more

Terraform Console and Output

Installing Terraform What is Terraform and Why do we need it? Terraform Commands Hashicorp Configuration Language - HCL Tainting and Untainting Resources We will use the Terraform Console to view various outputs that we can use for our scripts. The Terraform Console is extremely useful for troubleshooting and planning deployments. Terraform commands: console: Interactive console for Terraform interpolations Set up the environment: cd terraform/basics Working with the Terraform console Redeploy the Ghost image and container: terraform apply Show the Terraform resources: terraform show Start the Terraform console: terraform console Type the following in the console to get the container's name: docker_container.container_id.name Type the following in the console to get the container's IP: docker_container.container_id.ip_address Break out of the Terraform console by using Ctrl+C. Destroy the environment: terraform destroy Output the name and IP of the Ghost blog container Edit mai...
Read more

Terraform - Tainting and Updating Resources

Installing Terraform What is Terraform and Why do we need it? Terraform Commands Hashicorp Configuration Language - HCL We are going to take a look at how to force a redeploy of resources using tainting. This is an extremely useful skill for when parts of a deployment need to be modified. Tainting and Untainting Resources Terraform commands: taint: Manually mark a resource for recreation  untaint: Manually unmark a resource as tainted Tainting a resource: terraform taint [NAME] Untainting a resource: terraform untaint [NAME] Set up the environment: cd terraform/basics Redeploy the Ghost image: terraform apply Taint the Ghost blog resource: terraform taint docker_container.container_id See what will be changed: terraform plan Remove the taint on the Ghost blog resource: terraform untaint docker_container.container_id Verity that the Ghost blog resource is untainted: terraform plan Updating Resources Let's edit main.tf and change the image to ghost:alpine. Open main.tf: vi main.tf ma...
Read more