# week6-homework

**Repository Path**: zero-night-01/week6-homework

## Basic Information

- **Project Name**: week6-homework
- **Description**: No description available
- **Primary Language**: Scala
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 1
- **Forks**: 1
- **Created**: 2022-04-02
- **Last Updated**: 2022-05-22

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# week6-homework

#### 使用说明

题目一

思路：
1. 先对每个文件进行单词计数，此时 key 为文件名，value 为 (单词, 计数)
2. 对 key 和 value 重新整理即可得到倒排索引

核心代码

```scala
filesRDD
  // word count
  .flatMapValues(item => item.split("\n").flatMap(s => s.split("\\W+")))
  .map(contentToWordOne)
  // pay attention to skew due to large files
  .reduceByKey(_ + _)
  .cache()
  .map(item => (item._1._1, (item._1._2, item._2)))
  // pay attention to skew due to file nums
  .aggregateByKey(mutable.Map[String, Int]())(seqOp = aggregateIndex, combOp = combineIndex)
  .cache()
  // for test
  .collect()
  .sortBy(item => item._1)
  .foreach(printResult)
```

提交任务

```shell
spark-submit \
--class org.geektime.time.exercise1.InvertedIndexDemo \
--master local[*] \
file:///home/student1/fuwuchen/week6-homework-1.0-SNAPSHOT.jar \
appName \
local[*] \
file:///home/student1/fuwuchen/input/week6
```

示例输入

file0
```text
it is what it is
```

file1
```text
what is it
```

file2
```text
it is a banana
```

示例输出

```text
(a,{(file2,1)})
(banana,{(file2,1)})
(is,{(file2,1), (file1,1), (file0,2)})
(it,{(file2,1), (file1,1), (file0,2)})
(what,{(file1,1), (file0,1)})
```

题目二

```shell
Usage: copy directory from source fs to target fs
       for example: from hdfs://nn1:9000/test to hdfs://nn2:9000/test1
           will be: hdfs://nn2:9000/test1/test.
--source     :hdfs source path to copy, required.
--target     :hdfs target path, required.
-i           :ignore failures, value: true/false, default is true.
-m           :max concurrency, num of executors, default is 4.
--appName    :spark appName, default is SparkDistCpDemo.
--master     :spark master, default is local[*].
```

提交作业

```shell
spark-submit \
--class org.geektime.time.exercise2.SparkDistCpDemo \
--deploy-mode cluster \
--master yarn \
file:///home/student1/fuwuchen/week6-homework-1.0-SNAPSHOT.jar \
--source hdfs://emr-header-1.cluster-285604:9000/user/fuwuchen/week4 \
--target hdfs://emr-header-1.cluster-285604:9000/user/fuwuchen/week6 \
-i true \
-m 4 \
--appName SparkDistCpDemo \
--master yarn
```

示例输出
```text
22/04/17 10:47:42 INFO [task-result-getter-1] TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 343 ms on emr-header-1.cluster-285604 (executor driver) (2/4)
22/04/17 10:47:42 INFO [task-result-getter-2] TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 344 ms on emr-header-1.cluster-285604 (executor driver) (3/4)
22/04/17 10:47:43 INFO [Executor task launch worker for task 2.0 in stage 0.0 (TID 2)] Executor: Finished task 2.0 in stage 0.0 (TID 2). 880 bytes result sent to driver
22/04/17 10:47:43 INFO [task-result-getter-3] TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 410 ms on emr-header-1.cluster-285604 (executor driver) (4/4)
```

![WFQu5A](https://testpicp.oss-cn-shanghai.aliyuncs.com/uPic/WFQu5A.png)

可以看到并行度设置有效

备注：[YARN 作业地址](https://knox.c-01b94588f59c7655.cn-hangzhou.emr.aliyuncs.com:8443/gateway/cluster-topo/yarn/cluster/app/application_1645699879292_2321)