Single Node Hadoop Tutorial

Hands-on lab on Hadoop Map-Reduce

Objectives

  • Run a single-node Hadoop instance
  • Perform a word count using Hadoop Map Reduce.

Set up Single-Node Hadoop

The steps outlined in this lab use the single-node Hadoop Version 3.2.3. Hadoop is most useful when deployed in a fully distributed mode on a large cluster of networked servers sharing a large volume of data. However, for basic understanding, we will configure Hadoop on a single node.

In this lab, we will run the WordCount example with an input text and see how the content of the input file is processed by WordCount.

  1. Start a new terminal
  1. Download hadoop-3.2.3.tar.gz to your theia environment by running the following command.
curl https://dlcdn.apache.org/hadoop/common/hadoop-3.2.3/hadoop-3.2.3.tar.gz --output hadoop-3.2.3.tar.gz

    1. Extract the tar file in the currently directory.
    tar -xvf hadoop-3.2.3.tar.gz

    1. Navigate to the hadoop-3.2.3 directory.
    cd hadoop-3.2.3

    1. Check the hadoop command to see if it is setup. This will display the usage documentation for the hadoop script.
    bin/hadoop

    1. Run the following command to download data.txt to your current directory.
    curl https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-BD0225EN-SkillsNetwork/labs/data/data.txt --output data.txt

      1. Run the Map reduce application for wordcount on data.txt and store the output in /user/root/output
      bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.3.jar wordcount data.txt output

      This may take some time.

      1. Once the word count runs successfully, you can run the following command to see the output file it has generated.
      ls output

      You should see part-r-00000 with _SUCCESS indicating that the wordcount has been done.

      While it is still processing, you may only see '_temporary' listed in the output directory. Wait for a couple of minutes and run the command again till you see output as shown above.

      1. Run the following command to see the word count output.
      cat output/part-r-00000

      Local wordcount output

      The image below shows how the MapReduce wordcount happens.

      Practice Lab

      1. Do a word count on a file with the following content.
      India Italy Venice Italy Pizza India Pizza Pasta Gelato

       

      Do a word count on a file with very large content, try to find any text file that is in GBs and observe it.

       



      Comments

      Post a Comment

      Popular posts from this blog

      Terraform Commands - Hands-on Learning

      Installing Terraform What is Terraform and Why do we need it? Terraform Commands:- Here, we begin working with Terraform commands. We will start by creating a very simple Terraform file that will pull down the an image from Docker Hub. List the Terraform commands: terraform Common commands: apply: Builds or changes infrastructure console: Interactive console for Terraform interpolations destroy: Destroys Terraform-managed infrastructure fmt: Rewrites configuration files to canonical format get: Downloads and installs modules for the configuration graph: Creates a visual graph of Terraform resources import: Imports existing infrastructure into Terraform init: Initializes a new or existing Terraform configuration output: Reads an output from a state file plan: Generates and shows an execution plan providers: Prints a tree of the providers used in the configuration push: Uploads this Terraform module to Terraform Enterprise to run refresh: Updates local state file against real resources s...
      Read more

      Terraform Console and Output

      Installing Terraform What is Terraform and Why do we need it? Terraform Commands Hashicorp Configuration Language - HCL Tainting and Untainting Resources We will use the Terraform Console to view various outputs that we can use for our scripts. The Terraform Console is extremely useful for troubleshooting and planning deployments. Terraform commands: console: Interactive console for Terraform interpolations Set up the environment: cd terraform/basics Working with the Terraform console Redeploy the Ghost image and container: terraform apply Show the Terraform resources: terraform show Start the Terraform console: terraform console Type the following in the console to get the container's name: docker_container.container_id.name Type the following in the console to get the container's IP: docker_container.container_id.ip_address Break out of the Terraform console by using Ctrl+C. Destroy the environment: terraform destroy Output the name and IP of the Ghost blog container Edit mai...
      Read more

      Terraform - Tainting and Updating Resources

      Installing Terraform What is Terraform and Why do we need it? Terraform Commands Hashicorp Configuration Language - HCL We are going to take a look at how to force a redeploy of resources using tainting. This is an extremely useful skill for when parts of a deployment need to be modified. Tainting and Untainting Resources Terraform commands: taint: Manually mark a resource for recreation  untaint: Manually unmark a resource as tainted Tainting a resource: terraform taint [NAME] Untainting a resource: terraform untaint [NAME] Set up the environment: cd terraform/basics Redeploy the Ghost image: terraform apply Taint the Ghost blog resource: terraform taint docker_container.container_id See what will be changed: terraform plan Remove the taint on the Ghost blog resource: terraform untaint docker_container.container_id Verity that the Ghost blog resource is untainted: terraform plan Updating Resources Let's edit main.tf and change the image to ghost:alpine. Open main.tf: vi main.tf ma...
      Read more