# course-bigdata-analytics

**Repository Path**: houda-elfounti/course-bigdata-analytics

## Basic Information

- **Project Name**: course-bigdata-analytics
- **Description**: No description available
- **Primary Language**: Python
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 10
- **Created**: 2026-11-06
- **Last Updated**: 2026-11-06

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Big Data Technology and Application — Course Repository

[![Big Data](https://img.shields.io/badge/Big%20Data-Technology%20%26%20Application-2F6FEB?style=flat-square)](#course-overview) [![Analytics](https://img.shields.io/badge/Analytics-Data%20Science-6F42C1?style=flat-square)](#course-overview) [![Distributed Computing](https://img.shields.io/badge/Distributed-Computing-167D9A?style=flat-square)](#lecture-roadmap) [![Data Lake](https://img.shields.io/badge/Data%20Lake-Lakehouse-2E8B57?style=flat-square)](#lecture-roadmap) [![AI Data Services](https://img.shields.io/badge/AI-Data%20Services-FF8C42?style=flat-square)](#lecture-roadmap) [![License: CC BY-SA 4.0](https://img.shields.io/badge/License-CC%20BY--SA%204.0-lightgrey?style=flat-square)](LICENSE)

This repository contains the teaching materials, companion technical exercises, and project labs for **Big Data Technology and Application**. The course is designed for undergraduate computing students and connects big data engineering, data science, and AI data services.

## Course Overview

The course follows three connected technical strands:

| Strand | What students learn |
|---|---|
| **Data computing and analysis** | Understand distributed batch and stream processing; define data questions and metrics; explore data, compare baselines, interpret results, and communicate patterns with charts. |
| **Data development and management** | Ingest, store, transform, model, validate, govern, and deliver traceable and reproducible data assets. |
| **Data applications and visualization** | Present analysis through reports and visualizations; build and evaluate retrieval and AI-enabled data services, including vector search and retrieval-augmented generation (RAG). |

These strands meet in a continuous learning path: students frame a question, understand and prepare data, compute and analyze results, validate their interpretation, and deliver the outcome through an application. Big data engineering provides techniques for scalable data processing; data science contributes methods for framing questions and checking conclusions; AI extends the ways data can be served. They support one another without replacing one another.

The course comprises **16 lecture chapters (32 hours)** and **five project labs (20 hours)**, for **52 in-class hours**. Students also complete 40 hours of independent study. The five labs build one project incrementally, ending with a complete, interactive data application with a visualization dashboard.

## Learning Outcomes

By the end of the course, students should be able to:

1. Explain core distributed data system ideas, including partitioning, parallelism, fault tolerance, consistency, and batch and stream processing.
2. Select and use data technologies to build traceable, reproducible pipelines and governed data assets.
3. Frame data questions, inspect data, define metrics, compare a baseline, and explain results and their limitations.
4. Integrate data processing, analysis, retrieval, and visualization into an interactive application, and evaluate its behavior and evidence.

## Lecture Roadmap

| Chapters | Topics |
|---|---|
| 1–2 | From big data to data science; data lifecycle and the modern data stack |
| 3–5 | Distributed file and object storage; partitioning, replication, consistency, consensus, and recovery |
| 6–8 | NoSQL and HBase; MapReduce; Spark architecture, RDDs, and Spark SQL |
| 9–10 | Kafka and data ingestion; Flink, event time, and unified batch and stream processing |
| 11–13 | Data warehouse modeling and metrics; lakehouse table formats; data governance, quality, and privacy |
| 14–16 | Unstructured data and embeddings; vector retrieval and RAG; knowledge graphs and GraphRAG |

### Available Lecture Materials

| Chapter | Topic | Lecture file | PPT slides |
|---|---|---|---|
| 00 | Course introduction | [Markdown](ch00-course-introduction.md) | [PPT](ch00-course-introduction.pptx) |
| 01 | Data view | [Markdown](02-lectures/notes/ch01-data-view.md) | [PPT](ch01-data-view.pptx) |
| 02 | Technology stack | [Markdown](ch02-tech-stack.md) | [Part 1](ch02-tech-stack-part1.pptx) · [Part 2](ch02-tech-stack-part2.pptx) · [Part 3](ch02-tech-stack-part3.pptx) |

## Project Labs

L01–L05 use a shared project so each stage builds on earlier work:

| Lab | Focus | Main deliverable |
|---|---|---|
| **L01** | Frame the project question, explore data, establish metrics and a baseline, and create the first data pipeline | Problem and metric brief, data profile, runnable pipeline, and baseline |
| **L02** | Batch ETL, data quality, and analysis-ready datasets | ETL workflow, quality report, metric comparison, and documented data definitions |
| **L03** | Event stream processing, data integration, and time-series analysis | Event-processing workflow and comparison of streaming and batch results |
| **L04** | Vectorization, semantic retrieval, and evidence evaluation | Retrieval module, fixed evaluation set, and baseline report |
| **L05** | Integrate the project into a complete data application | Interactive application, visualization dashboard, evaluation, and reproducibility notes |

The chapter companion exercises cover focused technology setup and operation, including single-host, multi-container deployment with Docker Compose where applicable. The project labs focus on developing and integrating the application; they do not require a real multi-machine network.

## Repository Map

```text
course-bigdata-analytics/
├── 01-syllabus/        # Syllabus and detailed chapter outline (not yet added)
├── 02-lectures/        # Lecture notes, slides, diagrams, and exercises
│   ├── cn/             # Chinese materials (not yet added)
│   └── en/
│       ├── notes/      # Lecture notes for chapters 00–02
│       ├── slides/     # PPTX decks and diagram sources
│       └── tutorials/  # Chapter companion exercises and sample code
├── 03-colabs/          # Hands-on lab packages
│   └── colab-01-dataprep-mvp/
└── 04-projects/        # Student project and assignment submissions (see its README)
```

## Getting Started

- Read the [course introduction note](ch00-course-introduction.md) for the course structure, then the [Chapter 1 data view](02-lectures/notes/ch01-data-view.md) and [Chapter 2 technology stack](ch02-tech-stack.md) notes.
- Work through the chapter companion exercises: [Chapter 1 lab](02-lectures/tutorials/ch01/ch01-data-view.md) and [Chapter 2 lab](ch02-tech-stack-selection.md).
- Run the first hands-on lab from the [colab-01 START-HERE guide](03-colabs/colab-01-dataprep-mvp/START-HERE.md).
- The syllabus, detailed chapter outline, and the L01–L05 project lab guides are not yet in this repository.

## License

This README is licensed under [Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/). Copyright © 2025–2026 Gitconomy Research Community.