HDFS Architecture and Programming

Seize the savings! Get 40% off 3 months of Coursera Plus and full access to thousands of courses.

HDFS Architecture and Programming

This course is part of Big Data Processing Using Hadoop Specialization

Instructor: Karthik Shyamsunder

Included with

Learn more

4 modules

Gain insight into a topic and learn the fundamentals.

Intermediate level

Recommended experience

1 week to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

4 modules

Gain insight into a topic and learn the fundamentals.

Intermediate level

Recommended experience

1 week to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

What you'll learn

Understand HDFS architecture, components, and how it ensures scalability and availability for big data processing.
Learn to configure Hadoop for Java programming and perform file CRUD operations using HDFS APIs.
Master advanced HDFS programming concepts like compression, serialization, and working with specialized file structures like Sequence and Map files.

Skills you'll gain

Tools you'll learn

Apache Hadoop

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

9 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your subject-matter expertise

This course is part of the Big Data Processing Using Hadoop Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

There are 4 modules in this course

The course “HDFS Architecture and Programming” offers a comprehensive understanding of the Hadoop Distributed File System (HDFS) architecture, components, and advanced programming techniques. You will gain practical experience in setting up and configuring Hadoop for Java development, while mastering key concepts such as file and directory CRUD operations, data compression, and serialization. By the end of the course, you will be proficient in using HDFS to handle large-scale data processing, enabling you to build scalable, high-availability solutions.

What sets this course apart is its hands-on approach, where you will work directly with HDFS, writing client programs and applying advanced techniques such as using Sequence and Map Files for specialized data storage. Whether you're new to Hadoop or looking to refine your existing skills, this course equips you with the tools and knowledge to become proficient in HDFS programming, making you a valuable asset in the field of Big Data.

This course provides a comprehensive understanding of Hadoop Distributed File System (HDFS) architecture and its key components. Students will gain hands-on experience with HDFS, learning how to set up Java programming environments and configure Hadoop. The course covers essential topics such as the HDFS programming model, file and directory CRUD operations, and compression techniques. You will also explore serialization, deserialization, and specialized file structures like Sequence and Map Files. By the end of the course, You will be equipped to leverage HDFS for scalable, highly available big data solutions.

What's included

2 readings

In this module, we will cover the working model and architecture behind Hadoop Distributed File System (HDFS) 1.0 and the capabilities and deficiencies of HDFS 1.0 architecture.

What's included

6 videos4 readings3 assignments

6 videos Total 103 minutes

Introduction - HDFS Architecture 2 minutes
HDFS Architecture 33 minutes
Hands-on HDFS Architecture 33 minutes
Block Size, Replication, and Rack Awareness 17 minutes
HDFS Architectural Capabilities 17 minutes
Summary - HDFS Architecture 1 minute

4 readings Total 85 minutes

HDFS Architecture 15 minutes
Block Size, Replication, and Rack Awareness 15 minutes
HDFS Architectural Capabilities 15 minutes
Self-Reflective Reading: Comparing Hadoop's HDFS and MapR's FileSystem 40 minutes

3 assignments Total 90 minutes

HDFS Architecture 60 minutes
Mastering HDFS: Architecture Overview and Hands-On Experience 15 minutes
Deep Dive into HDFS: Block Size, Replication, and Architectural Capabilities 15 minutes

In this module, we will cover HDFS programming concepts, HDFS API, and steps to write an HDFS client program for CRUD (Create, Read, Update and Delete) on files.

What's included

6 videos5 readings3 assignments

6 videos Total 106 minutes

Introduction - HDFS Programming Basics 2 minutes
HDFS C API Overview 11 minutes
Hadoop Configuration API and Dev Env Setup 38 minutes
HDFS Java API Overview 19 minutes
File CRUD 34 minutes
Summary - HDFS Programming Basics 2 minutes

5 readings Total 105 minutes

HDFS C API Overview 15 minutes
Hadoop Configuration API and Dev Env Setup 20 minutes
HDFS Java API Overview 15 minutes
File CRUD 15 minutes
Self-Reflective Reading: Design Reflection: Error Handling in File Deletion 40 minutes

3 assignments Total 90 minutes

HDFS Programming Basics 60 minutes
Getting Started with HDFS Programming: Basics, C API, and Configuration 15 minutes
HDFS Programming Essentials: Java API, File CRUD Operations, and Summary 15 minutes

In this module, we will cover HDFS advanced programming concepts, such as CRUD on directories, compression, serialization and deserialization, and file-based data structures like sequence files.

What's included

6 videos5 readings3 assignments

6 videos Total 77 minutes

Introduction - HDFS Advanced Programming 2 minutes
Directory CRUD 14 minutes
File Compression Decompression 20 minutes
Serialization Deserialization 18 minutes
File-Based Data Structures 21 minutes
Summary - HDFS Advanced Programming 3 minutes

5 readings Total 80 minutes

Directory CRUD 15 minutes
File Compression Decompression 10 minutes
Serialization Deserialization 10 minutes
File-Based Data Structures 15 minutes
Self-Reflective Reading: Hadoop's Serialization Decision 30 minutes

3 assignments Total 90 minutes

HDFS Programming Advanced 60 minutes
Advanced HDFS Programming: Directory CRUD and File Compression/Decompression 15 minutes
Advanced HDFS Programming: Serialization, File-Based Data Structures, and Summary 15 minutes

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV. Share it on social media and in your performance review.

Instructor

Karthik Shyamsunder

Johns Hopkins University

4 Courses 1,277 learners

Offered by

Johns Hopkins University

Explore more from Data Management

Status: Free Trial
Johns Hopkins University
YARN MapReduce Architecture and Advanced Programming
Course
Status: Free Trial
EDUCBA
Hadoop: Analyze, Configure & Manage Big Data
Course
University of California San Diego
Hadoop Platform and Application Framework
Course
Status: Free Trial
Pearson
Hadoop and Spark Fundamentals: Unit 1
Course

Why people choose Coursera for their career

Felipe M.

Learner since 2018

"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."

Jennifer J.

Learner since 2020

"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."

Larry W.

Learner since 2021

"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."

Chaitanya A.

"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."

Open new doors with Coursera Plus

Unlimited access to 10,000+ world-class courses, hands-on projects, and job-ready certificate programs - all included in your subscription

Learn more

Advance your career with an online degree

Earn a degree from world-class universities - 100% online

Explore degrees

Join over 3,400 global companies that choose Coursera for Business

Upskill your employees to excel in the digital economy

Learn more

Frequently asked questions

To access the course materials, assignments and to earn a Certificate, you will need to purchase the Certificate experience when you enroll in a course. You can try a Free Trial instead, or apply for Financial Aid. The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.

When you enroll in the course, you get access to all of the courses in the Specialization, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile.

Yes. In select learning programs, you can apply for financial aid or a scholarship if you can’t afford the enrollment fee. If fin aid or scholarship is available for your learning program selection, you’ll find a link to apply on the description page.