Integrating LVM with Hadoop and providing Elasticity to DataNode
- Hadoop installed
- Two disks
Elasticity to Datanode means Increase and decrease the DataNode on the fly or dynamically.
The way to create LVM goes as below:
Step 1: Create physical volume.
Step 2: Create volume group.
vgcreate myvg /dev/xvdg
Step 3: Create LVM.
lvcreate --size 2G --name lv1 myvg
Step 4: format the LVM partition
- we are formating the lv1 partition with ext4 filesystem.
Step 5:mount the LVM partition
mount /dev/myvg/lv1 /dn
- Here /dn is the mount point and this mount point will be the storage
directory in the hdfs-site.xml file of the hadoop.
- Start the hadoop services as we are going to share the datanode storage in the hadoop hdfs cluster.
Now it is time to show the elasticity of datanode storage:
Extend the LVM as follows:
lvextend --size +1.5G /dev/myvg/lv1
LVM does not stop at increasing or decreasing the partition, even it can add storage from the another disk by adding the disk as physical volume in the LVM’S volume group.
Steps to do so are as follows:
Step 1: list all the block storage
Step 2: Create the physical volume
Step 3: Extend the volume group
vgextend myvg /dev/xvdf
Step 4: Extend the LVM partition
lvextend --size +5G /dev/myvg/lv1
Conclusion: I started from sharing the 2G storage to the hdfs cluster and then increase the storage till 4G. After sharing the 4G i show how we can merge the storage of two disks to provide the large storage to the hdfs cluster.