Integrating LVM with Hadoop and providing Elasticity to DataNodeStorage

prerequisite:

• Hadoop installed
• Two disks

Elasticity to Datanode means Increase and decrease the DataNode on the fly or dynamically.

The way to create LVM goes as below:

Step 1: Create physical volume.

pvcreate /dev/xvdg

Step 2: Create volume group.

vgcreate myvg /dev/xvdg

Step 3: Create LVM.

lvcreate --size 2G --name lv1 myvg

Step 4: format the LVM partition

mkfs.ext4 /dev/myvg/lv1

• we are formating the lv1 partition with ext4 filesystem.

Step 5:mount the LVM partition

mount /dev/myvg/lv1 /dn

• Here /dn is the mount point and this mount point will be the storage

directory in the hdfs-site.xml file of the hadoop.

• Start the hadoop services as we are going to share the datanode storage in the hadoop hdfs cluster.

Hadoop hdfs cluster report

Now it is time to show the elasticity of datanode storage:

Extend the LVM as follows:

lvextend --size +1.5G /dev/myvg/lv1

Hadoop hdfs cluster report

LVM does not stop at increasing or decreasing the partition, even it can add storage from the another disk by adding the disk as physical volume in the LVM’S volume group.

Steps to do so are as follows:

Step 1: list all the block storage

lsblk

Step 2: Create the physical volume

pvcreate /dev/xvdf

Step 3: Extend the volume group

vgextend myvg /dev/xvdf

Step 4: Extend the LVM partition

lvextend --size +5G /dev/myvg/lv1

Conclusion: I started from sharing the 2G storage to the hdfs cluster and then increase the storage till 4G. After sharing the 4G i show how we can merge the storage of two disks to provide the large storage to the hdfs cluster.