如何利用MapReduce的分治算法策略提高KNN算法的运行速度

4035人阅读
Machine Learning(6)
集群环境介绍:
hadoop2.4.1 64位
6台服务器:
NameNode 、SecondaryNameNode
ResourceManager
DataNode、NodeManager
DataNode、NodeManager
DataNode、NodeManager
DataNode、NodeManager
实验1:训练集train.txt样例个数为.24M)
测试集test.txt样例个数为5kb),并将全部测试集都存放在test.txt中
[root@hadoop11 local]
-rw-r--r--
3 root supergroup
3400816 2016-07-17 19:28 /dir6/test.txt
注意:此时所有的测试集都在一个文本中(test.txt)存放,作为输入路径
KNN算法运行日志:
16/07/17 19:32:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop22/10.187.84.51:8032
16/07/17 19:32:25 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/07/17 19:32:25 INFO input.FileInputFormat: Total input paths to process : 1
16/07/17 19:32:25 INFO mapreduce.JobSubmitter: number of splits:1
16/07/17 19:32:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_5_0016
16/07/17 19:32:26 INFO impl.YarnClientImpl: Submitted application application_5_0016
16/07/17 19:32:26 INFO mapreduce.Job: The url to track the job: http:
16/07/17 19:32:26 INFO mapreduce.Job: Running job: job_5_0016
16/07/17 19:32:32 INFO mapreduce.Job: Job job_5_0016 running in uber mode : false
16/07/17 19:32:32 INFO mapreduce.Job:
map 0% reduce 0%
16/07/17 19:32:49 INFO mapreduce.Job:
map 1% reduce 0%
16/07/17 19:33:05 INFO mapreduce.Job:
map 2% reduce 0%
16/07/17 19:33:20 INFO mapreduce.Job:
map 3% reduce 0%
16/07/17 19:33:35 INFO mapreduce.Job:
map 4% reduce 0%
16/07/17 19:33:50 INFO mapreduce.Job:
map 5% reduce 0%
16/07/17 19:34:02 INFO mapreduce.Job:
map 6% reduce 0%
16/07/17 19:34:17 INFO mapreduce.Job:
map 7% reduce 0%
16/07/17 19:34:32 INFO mapreduce.Job:
map 8% reduce 0%
16/07/17 19:34:47 INFO mapreduce.Job:
map 9% reduce 0%
16/07/17 19:35:02 INFO mapreduce.Job:
map 10% reduce 0%
16/07/17 19:35:14 INFO mapreduce.Job:
map 11% reduce 0%
16/07/17 19:35:29 INFO mapreduce.Job:
map 12% reduce 0%
16/07/17 19:35:44 INFO mapreduce.Job:
map 13% reduce 0%
16/07/17 19:35:59 INFO mapreduce.Job:
map 14% reduce 0%
16/07/17 19:36:12 INFO mapreduce.Job:
map 15% reduce 0%
16/07/17 19:36:27 INFO mapreduce.Job:
map 16% reduce 0%
16/07/17 19:36:42 INFO mapreduce.Job:
map 17% reduce 0%
16/07/17 19:36:57 INFO mapreduce.Job:
map 18% reduce 0%
16/07/17 19:37:12 INFO mapreduce.Job:
map 19% reduce 0%
16/07/17 19:37:27 INFO mapreduce.Job:
map 20% reduce 0%
16/07/17 19:37:39 INFO mapreduce.Job:
map 21% reduce 0%
16/07/17 19:37:54 INFO mapreduce.Job:
map 22% reduce 0%
16/07/17 19:38:09 INFO mapreduce.Job:
map 23% reduce 0%
16/07/17 19:38:24 INFO mapreduce.Job:
map 24% reduce 0%
16/07/17 19:38:39 INFO mapreduce.Job:
map 25% reduce 0%
16/07/17 19:38:51 INFO mapreduce.Job:
map 26% reduce 0%
16/07/17 19:39:06 INFO mapreduce.Job:
map 27% reduce 0%
16/07/17 19:39:22 INFO mapreduce.Job:
map 28% reduce 0%
16/07/17 19:39:37 INFO mapreduce.Job:
map 29% reduce 0%
16/07/17 19:39:52 INFO mapreduce.Job:
map 30% reduce 0%
16/07/17 19:40:07 INFO mapreduce.Job:
map 31% reduce 0%
16/07/17 19:40:22 INFO mapreduce.Job:
map 32% reduce 0%
16/07/17 19:40:37 INFO mapreduce.Job:
map 33% reduce 0%
16/07/17 19:40:52 INFO mapreduce.Job:
map 34% reduce 0%
16/07/17 19:41:04 INFO mapreduce.Job:
map 35% reduce 0%
16/07/17 19:41:22 INFO mapreduce.Job:
map 36% reduce 0%
16/07/17 19:41:37 INFO mapreduce.Job:
map 37% reduce 0%
16/07/17 19:41:52 INFO mapreduce.Job:
map 38% reduce 0%
16/07/17 19:42:07 INFO mapreduce.Job:
map 39% reduce 0%
16/07/17 19:42:22 INFO mapreduce.Job:
map 40% reduce 0%
16/07/17 19:42:37 INFO mapreduce.Job:
map 41% reduce 0%
16/07/17 19:42:53 INFO mapreduce.Job:
map 42% reduce 0%
16/07/17 19:43:08 INFO mapreduce.Job:
map 43% reduce 0%
16/07/17 19:43:23 INFO mapreduce.Job:
map 44% reduce 0%
16/07/17 19:43:41 INFO mapreduce.Job:
map 45% reduce 0%
16/07/17 19:43:56 INFO mapreduce.Job:
map 46% reduce 0%
16/07/17 19:44:12 INFO mapreduce.Job:
map 47% reduce 0%
16/07/17 19:44:30 INFO mapreduce.Job:
map 48% reduce 0%
16/07/17 19:44:45 INFO mapreduce.Job:
map 49% reduce 0%
16/07/17 19:45:00 INFO mapreduce.Job:
map 50% reduce 0%
16/07/17 19:45:15 INFO mapreduce.Job:
map 51% reduce 0%
16/07/17 19:45:30 INFO mapreduce.Job:
map 52% reduce 0%
16/07/17 19:45:48 INFO mapreduce.Job:
map 53% reduce 0%
16/07/17 19:46:03 INFO mapreduce.Job:
map 54% reduce 0%
16/07/17 19:46:18 INFO mapreduce.Job:
map 55% reduce 0%
16/07/17 19:46:33 INFO mapreduce.Job:
map 56% reduce 0%
16/07/17 19:46:49 INFO mapreduce.Job:
map 57% reduce 0%
16/07/17 19:47:07 INFO mapreduce.Job:
map 58% reduce 0%
16/07/17 19:47:22 INFO mapreduce.Job:
map 59% reduce 0%
16/07/17 19:47:37 INFO mapreduce.Job:
map 60% reduce 0%
16/07/17 19:47:55 INFO mapreduce.Job:
map 61% reduce 0%
16/07/17 19:48:10 INFO mapreduce.Job:
map 62% reduce 0%
16/07/17 19:48:25 INFO mapreduce.Job:
map 63% reduce 0%
16/07/17 19:48:43 INFO mapreduce.Job:
map 64% reduce 0%
16/07/17 19:48:58 INFO mapreduce.Job:
map 65% reduce 0%
16/07/17 19:49:13 INFO mapreduce.Job:
map 66% reduce 0%
16/07/17 19:49:28 INFO mapreduce.Job:
map 67% reduce 0%
16/07/17 19:49:30 INFO mapreduce.Job:
map 100% reduce 0%
16/07/17 19:49:37 INFO mapreduce.Job:
map 100% reduce 100%
16/07/17 19:49:38 INFO mapreduce.Job: Job job_5_0016 completed successfully
16/07/17 19:49:39 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2892255
FILE: Number of bytes written=5971253
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=4056338
HDFS: Number of bytes written=861195
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=1016177
Total time spent by all reduces in occupied slots (ms)=4948
Total time spent by all map tasks (ms)=1016177
Total time spent by all reduce tasks (ms)=4948
Total vcore-seconds taken by all map tasks=1016177
Total vcore-seconds taken by all reduce tasks=4948
Total megabyte-seconds taken by all map tasks=
Total megabyte-seconds taken by all reduce tasks=5066752
Map-Reduce Framework
Map input records=51444
Map output records=154332
Map output bytes=2583585
Map output materialized bytes=2892255
Input split bytes=103
Combine input records=0
Combine output records=0
Reduce input groups=51444
Reduce shuffle bytes=2892255
Reduce input records=154332
Reduce output records=51444
Spilled Records=308664
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=5836
CPU time spent (ms)=1033510
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=655419
File Output Format Counters
Bytes Written=861195
精确度:51444
CPU time spent (ms)=1033510
map tasks=1
实验2:训练集train.txt样例个数为245057不变
测试集test.txt样例个数为51444,并将全部测试集存放在
test1.txt(25568)和test2.txt(25857)中
KNN算法运行日志:
先看进程日志:
[root@hadoop66 ~]
24659 YarnChild
(mapper任务)
22777 DataNode
24660 YarnChild
(mapper任务)
24557 MRAppMaster
22622 NodeManager
计数器日志:
[root@hadoop11 local]# app1.sh
16/07/17 20:21:03 INFO client.RMProxy: Connecting to ResourceManager at hadoop22/10.187.84.51:8032
16/07/17 20:21:03 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/07/17 20:21:03 INFO input.FileInputFormat: Total input paths to process : 2
16/07/17 20:21:03 INFO mapreduce.JobSubmitter: number of splits:2
16/07/17 20:21:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_5_0019
16/07/17 20:21:04 INFO impl.YarnClientImpl: Submitted application application_5_0019
16/07/17 20:21:04 INFO mapreduce.Job: The url to track the job: http://hadoop22:8088/proxy/application_5_0019/
16/07/17 20:21:04 INFO mapreduce.Job: Running job: job_5_0019
16/07/17 20:21:10 INFO mapreduce.Job: Job job_5_0019 running in uber mode : false
16/07/17 20:21:10 INFO mapreduce.Job:
map 0% reduce 0%
16/07/17 20:21:21 INFO mapreduce.Job:
map 1% reduce 0%
16/07/17 20:21:30 INFO mapreduce.Job:
map 2% reduce 0%
16/07/17 20:21:40 INFO mapreduce.Job:
map 3% reduce 0%
16/07/17 20:21:46 INFO mapreduce.Job:
map 4% reduce 0%
16/07/17 20:21:55 INFO mapreduce.Job:
map 5% reduce 0%
16/07/17 20:22:01 INFO mapreduce.Job:
map 6% reduce 0%
16/07/17 20:22:10 INFO mapreduce.Job:
map 7% reduce 0%
16/07/17 20:22:17 INFO mapreduce.Job:
map 8% reduce 0%
16/07/17 20:22:26 INFO mapreduce.Job:
map 9% reduce 0%
16/07/17 20:22:35 INFO mapreduce.Job:
map 10% reduce 0%
16/07/17 20:22:41 INFO mapreduce.Job:
map 11% reduce 0%
16/07/17 20:22:47 INFO mapreduce.Job:
map 12% reduce 0%
16/07/17 20:22:56 INFO mapreduce.Job:
map 13% reduce 0%
16/07/17 20:23:05 INFO mapreduce.Job:
map 14% reduce 0%
16/07/17 20:23:11 INFO mapreduce.Job:
map 15% reduce 0%
16/07/17 20:23:17 INFO mapreduce.Job:
map 16% reduce 0%
16/07/17 20:23:26 INFO mapreduce.Job:
map 17% reduce 0%
16/07/17 20:23:35 INFO mapreduce.Job:
map 18% reduce 0%
16/07/17 20:23:41 INFO mapreduce.Job:
map 19% reduce 0%
16/07/17 20:23:50 INFO mapreduce.Job:
map 20% reduce 0%
16/07/17 20:23:56 INFO mapreduce.Job:
map 21% reduce 0%
16/07/17 20:24:05 INFO mapreduce.Job:
map 22% reduce 0%
16/07/17 20:24:11 INFO mapreduce.Job:
map 23% reduce 0%
16/07/17 20:24:20 INFO mapreduce.Job:
map 24% reduce 0%
16/07/17 20:24:26 INFO mapreduce.Job:
map 25% reduce 0%
16/07/17 20:24:35 INFO mapreduce.Job:
map 26% reduce 0%
16/07/17 20:24:42 INFO mapreduce.Job:
map 27% reduce 0%
16/07/17 20:24:51 INFO mapreduce.Job:
map 28% reduce 0%
16/07/17 20:24:57 INFO mapreduce.Job:
map 29% reduce 0%
16/07/17 20:25:06 INFO mapreduce.Job:
map 30% reduce 0%
16/07/17 20:25:12 INFO mapreduce.Job:
map 31% reduce 0%
16/07/17 20:25:21 INFO mapreduce.Job:
map 32% reduce 0%
16/07/17 20:25:27 INFO mapreduce.Job:
map 33% reduce 0%
16/07/17 20:25:36 INFO mapreduce.Job:
map 34% reduce 0%
16/07/17 20:25:42 INFO mapreduce.Job:
map 35% reduce 0%
16/07/17 20:25:51 INFO mapreduce.Job:
map 36% reduce 0%
16/07/17 20:25:57 INFO mapreduce.Job:
map 37% reduce 0%
16/07/17 20:26:06 INFO mapreduce.Job:
map 38% reduce 0%
16/07/17 20:26:12 INFO mapreduce.Job:
map 39% reduce 0%
16/07/17 20:26:21 INFO mapreduce.Job:
map 40% reduce 0%
16/07/17 20:26:30 INFO mapreduce.Job:
map 41% reduce 0%
16/07/17 20:26:36 INFO mapreduce.Job:
map 42% reduce 0%
16/07/17 20:26:45 INFO mapreduce.Job:
map 43% reduce 0%
16/07/17 20:26:51 INFO mapreduce.Job:
map 44% reduce 0%
16/07/17 20:27:00 INFO mapreduce.Job:
map 45% reduce 0%
16/07/17 20:27:06 INFO mapreduce.Job:
map 46% reduce 0%
16/07/17 20:27:15 INFO mapreduce.Job:
map 47% reduce 0%
16/07/17 20:27:21 INFO mapreduce.Job:
map 48% reduce 0%
16/07/17 20:27:30 INFO mapreduce.Job:
map 49% reduce 0%
16/07/17 20:27:36 INFO mapreduce.Job:
map 50% reduce 0%
16/07/17 20:27:45 INFO mapreduce.Job:
map 51% reduce 0%
16/07/17 20:27:51 INFO mapreduce.Job:
map 52% reduce 0%
16/07/17 20:28:01 INFO mapreduce.Job:
map 53% reduce 0%
16/07/17 20:28:07 INFO mapreduce.Job:
map 54% reduce 0%
16/07/17 20:28:16 INFO mapreduce.Job:
map 55% reduce 0%
16/07/17 20:28:23 INFO mapreduce.Job:
map 56% reduce 0%
16/07/17 20:28:31 INFO mapreduce.Job:
map 57% reduce 0%
16/07/17 20:28:38 INFO mapreduce.Job:
map 58% reduce 0%
16/07/17 20:28:46 INFO mapreduce.Job:
map 59% reduce 0%
16/07/17 20:28:53 INFO mapreduce.Job:
map 60% reduce 0%
16/07/17 20:29:02 INFO mapreduce.Job:
map 61% reduce 0%
16/07/17 20:29:10 INFO mapreduce.Job:
map 62% reduce 0%
16/07/17 20:29:17 INFO mapreduce.Job:
map 63% reduce 0%
16/07/17 20:29:26 INFO mapreduce.Job:
map 64% reduce 0%
16/07/17 20:29:32 INFO mapreduce.Job:
map 65% reduce 0%
16/07/17 20:29:41 INFO mapreduce.Job:
map 66% reduce 0%
16/07/17 20:29:42 INFO mapreduce.Job:
map 83% reduce 0%
16/07/17 20:29:52 INFO mapreduce.Job:
map 83% reduce 17%
16/07/17 20:29:54 INFO mapreduce.Job:
map 100% reduce 17%
16/07/17 20:29:55 INFO mapreduce.Job:
map 100% reduce 70%
16/07/17 20:29:56 INFO mapreduce.Job:
map 100% reduce 100%
16/07/17 20:29:56 INFO mapreduce.Job: Job job_5_0019 completed successfully
16/07/17 20:29:56 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2892255
FILE: Number of bytes written=6064619
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=7482816
HDFS: Number of bytes written=861195
HDFS: Number of read operations=11
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=1032086
Total time spent by all reduces in occupied slots (ms)=11757
Total time spent by all map tasks (ms)=1032086
Total time spent by all reduce tasks (ms)=11757
Total vcore-seconds taken by all map tasks=1032086
Total vcore-seconds taken by all reduce tasks=11757
Total megabyte-seconds taken by all map tasks=
Total megabyte-seconds taken by all reduce tasks=
Map-Reduce Framework
Map input records=51444
Map output records=154332
Map output bytes=2583585
Map output materialized bytes=2892261
Input split bytes=200
Combine input records=0
Combine output records=0
Reduce input groups=51444
Reduce shuffle bytes=2892261
Reduce input records=154332
Reduce output records=51444
Spilled Records=308664
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=8264
CPU time spent (ms)=1045670
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=680984
File Output Format Counters
Bytes Written=861195
16/07/17 20:29:58 INFO client.RMProxy: Connecting to ResourceManager at hadoop22/10.187.84.51:8032
16/07/17 20:29:59 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/07/17 20:29:59 INFO input.FileInputFormat: Total input paths to process : 1
16/07/17 20:29:59 INFO mapreduce.JobSubmitter: number of splits:1
16/07/17 20:29:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_5_0020
16/07/17 20:29:59 INFO impl.YarnClientImpl: Submitted application application_5_0020
16/07/17 20:30:00 INFO mapreduce.Job: The url to track the job: http://hadoop22:8088/proxy/application_5_0020/
16/07/17 20:30:00 INFO mapreduce.Job: Running job: job_5_0020
16/07/17 20:30:05 INFO mapreduce.Job: Job job_5_0020 running in uber mode : false
16/07/17 20:30:05 INFO mapreduce.Job:
map 0% reduce 0%
16/07/17 20:30:12 INFO mapreduce.Job:
map 100% reduce 0%
16/07/17 20:30:18 INFO mapreduce.Job:
map 100% reduce 100%
16/07/17 20:30:18 INFO mapreduce.Job: Job job_5_0020 completed successfully
16/07/17 20:30:18 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=24
FILE: Number of bytes written=186173
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=861298
HDFS: Number of bytes written=12
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3973
Total time spent by all reduces in occupied slots (ms)=3243
Total time spent by all map tasks (ms)=3973
Total time spent by all reduce tasks (ms)=3243
Total vcore-seconds taken by all map tasks=3973
Total vcore-seconds taken by all reduce tasks=3243
Total megabyte-seconds taken by all map tasks=4068352
Total megabyte-seconds taken by all reduce tasks=3320832
Map-Reduce Framework
Map input records=51444
Map output records=1
Map output bytes=16
Map output materialized bytes=24
Input split bytes=103
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=24
Reduce input records=1
Reduce output records=1
Spilled Records=2
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=70
CPU time spent (ms)=2340
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=861195
File Output Format Counters
Bytes Written=12
精确度:51444
CPU time spent (ms)=1045670
(时间之所以长:在于mapper任务的创建花费了时间,并且两个mapper任务都在同一个服务器hadoop66运行)
map tasks=2
实验3:训练集train.txt样例个数为245057不变
测试集test.txt样例个数为51444,并将全部测试集存放在
test1.txt(25402)和test2.txt(15224)和test3.txt(10818)中
先看进程日志:
[root@hadoop33 ~]
26279 YarnChild
(mapper任务)
2399 QuorumPeerMain
26280 YarnChild
(mapper任务)
23800 DataNode
23648 NodeManager
26133 MRAppMaster
[root@hadoop66 ~]
22777 DataNode
26302 YarnChild
(mapper任务)
22622 NodeManager
此时可以看出,此时mapper任务的执行有两台服务器来执行---分而治之!
具体运行日志:
[root@hadoop11 local]# app1.sh
16/07/17 20:55:17 INFO client.RMProxy: Connecting to ResourceManager at hadoop22/10.187.84.51:8032
16/07/17 20:55:18 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/07/17 20:55:18 INFO input.FileInputFormat: Total input paths to process : 3
16/07/17 20:55:18 INFO mapreduce.JobSubmitter: number of splits:3
16/07/17 20:55:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_5_0021
16/07/17 20:55:19 INFO impl.YarnClientImpl: Submitted application application_5_0021
16/07/17 20:55:19 INFO mapreduce.Job: The url to track the job: http://hadoop22:8088/proxy/application_5_0021/
16/07/17 20:55:19 INFO mapreduce.Job: Running job: job_5_0021
16/07/17 20:55:25 INFO mapreduce.Job: Job job_5_0021 running in uber mode : false
16/07/17 20:55:25 INFO mapreduce.Job:
map 0% reduce 0%
16/07/17 20:55:37 INFO mapreduce.Job:
map 1% reduce 0%
16/07/17 20:55:40 INFO mapreduce.Job:
map 2% reduce 0%
16/07/17 20:55:45 INFO mapreduce.Job:
map 3% reduce 0%
16/07/17 20:55:49 INFO mapreduce.Job:
map 4% reduce 0%
16/07/17 20:55:54 INFO mapreduce.Job:
map 5% reduce 0%
16/07/17 20:55:58 INFO mapreduce.Job:
map 6% reduce 0%
16/07/17 20:56:03 INFO mapreduce.Job:
map 7% reduce 0%
16/07/17 20:56:07 INFO mapreduce.Job:
map 8% reduce 0%
16/07/17 20:56:12 INFO mapreduce.Job:
map 9% reduce 0%
16/07/17 20:56:16 INFO mapreduce.Job:
map 10% reduce 0%
16/07/17 20:56:20 INFO mapreduce.Job:
map 11% reduce 0%
16/07/17 20:56:24 INFO mapreduce.Job:
map 12% reduce 0%
16/07/17 20:56:29 INFO mapreduce.Job:
map 13% reduce 0%
16/07/17 20:56:33 INFO mapreduce.Job:
map 14% reduce 0%
16/07/17 20:56:37 INFO mapreduce.Job:
map 15% reduce 0%
16/07/17 20:56:42 INFO mapreduce.Job:
map 16% reduce 0%
16/07/17 20:56:47 INFO mapreduce.Job:
map 17% reduce 0%
16/07/17 20:56:51 INFO mapreduce.Job:
map 18% reduce 0%
16/07/17 20:56:56 INFO mapreduce.Job:
map 19% reduce 0%
16/07/17 20:57:00 INFO mapreduce.Job:
map 20% reduce 0%
16/07/17 20:57:05 INFO mapreduce.Job:
map 21% reduce 0%
16/07/17 20:57:08 INFO mapreduce.Job:
map 22% reduce 0%
16/07/17 20:57:13 INFO mapreduce.Job:
map 23% reduce 0%
16/07/17 20:57:18 INFO mapreduce.Job:
map 24% reduce 0%
16/07/17 20:57:23 INFO mapreduce.Job:
map 25% reduce 0%
16/07/17 20:57:27 INFO mapreduce.Job:
map 26% reduce 0%
16/07/17 20:57:32 INFO mapreduce.Job:
map 27% reduce 0%
16/07/17 20:57:36 INFO mapreduce.Job:
map 28% reduce 0%
16/07/17 20:57:41 INFO mapreduce.Job:
map 29% reduce 0%
16/07/17 20:57:45 INFO mapreduce.Job:
map 30% reduce 0%
16/07/17 20:57:50 INFO mapreduce.Job:
map 31% reduce 0%
16/07/17 20:57:54 INFO mapreduce.Job:
map 32% reduce 0%
16/07/17 20:57:59 INFO mapreduce.Job:
map 33% reduce 0%
16/07/17 20:58:03 INFO mapreduce.Job:
map 34% reduce 0%
16/07/17 20:58:08 INFO mapreduce.Job:
map 35% reduce 0%
16/07/17 20:58:12 INFO mapreduce.Job:
map 36% reduce 0%
16/07/17 20:58:15 INFO mapreduce.Job:
map 37% reduce 0%
16/07/17 20:58:20 INFO mapreduce.Job:
map 38% reduce 0%
16/07/17 20:58:24 INFO mapreduce.Job:
map 39% reduce 0%
16/07/17 20:58:29 INFO mapreduce.Job:
map 40% reduce 0%
16/07/17 20:58:33 INFO mapreduce.Job:
map 41% reduce 0%
16/07/17 20:58:38 INFO mapreduce.Job:
map 42% reduce 0%
16/07/17 20:58:42 INFO mapreduce.Job:
map 43% reduce 0%
16/07/17 20:58:47 INFO mapreduce.Job:
map 44% reduce 0%
16/07/17 20:58:51 INFO mapreduce.Job:
map 45% reduce 0%
16/07/17 20:58:56 INFO mapreduce.Job:
map 46% reduce 0%
16/07/17 20:59:00 INFO mapreduce.Job:
map 58% reduce 0%
16/07/17 20:59:06 INFO mapreduce.Job:
map 59% reduce 0%
16/07/17 20:59:11 INFO mapreduce.Job:
map 59% reduce 11%
16/07/17 20:59:15 INFO mapreduce.Job:
map 60% reduce 11%
16/07/17 20:59:21 INFO mapreduce.Job:
map 61% reduce 11%
16/07/17 20:59:30 INFO mapreduce.Job:
map 62% reduce 11%
16/07/17 20:59:39 INFO mapreduce.Job:
map 63% reduce 11%
16/07/17 20:59:48 INFO mapreduce.Job:
map 64% reduce 11%
16/07/17 20:59:58 INFO mapreduce.Job:
map 65% reduce 11%
16/07/17 21:00:04 INFO mapreduce.Job:
map 66% reduce 11%
16/07/17 21:00:13 INFO mapreduce.Job:
map 67% reduce 11%
16/07/17 21:00:23 INFO mapreduce.Job:
map 68% reduce 11%
16/07/17 21:00:26 INFO mapreduce.Job:
map 79% reduce 11%
16/07/17 21:00:27 INFO mapreduce.Job:
map 79% reduce 22%
16/07/17 21:00:35 INFO mapreduce.Job:
map 80% reduce 22%
16/07/17 21:00:59 INFO mapreduce.Job:
map 81% reduce 22%
16/07/17 21:01:20 INFO mapreduce.Job:
map 82% reduce 22%
16/07/17 21:01:44 INFO mapreduce.Job:
map 83% reduce 22%
16/07/17 21:02:08 INFO mapreduce.Job:
map 84% reduce 22%
16/07/17 21:02:32 INFO mapreduce.Job:
map 85% reduce 22%
16/07/17 21:02:56 INFO mapreduce.Job:
map 86% reduce 22%
16/07/17 21:03:17 INFO mapreduce.Job:
map 87% reduce 22%
16/07/17 21:03:41 INFO mapreduce.Job:
map 88% reduce 22%
16/07/17 21:04:06 INFO mapreduce.Job:
map 89% reduce 22%
16/07/17 21:04:15 INFO mapreduce.Job:
map 100% reduce 22%
16/07/17 21:04:16 INFO mapreduce.Job:
map 100% reduce 90%
16/07/17 21:04:17 INFO mapreduce.Job:
map 100% reduce 100%
16/07/17 21:04:17 INFO mapreduce.Job: Job job_5_0021 completed successfully
16/07/17 21:04:17 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=2892255
FILE: Number of bytes written=6158011
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=
HDFS: Number of bytes written=861195
HDFS: Number of read operations=15
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Killed map tasks=2
Launched map tasks=5
Launched reduce tasks=1
Data-local map tasks=5
Total time spent by all maps in occupied slots (ms)=1417294
Total time spent by all reduces in occupied slots (ms)=313657
Total time spent by all map tasks (ms)=1417294
Total time spent by all reduce tasks (ms)=313657
Total vcore-seconds taken by all map tasks=1417294
Total vcore-seconds taken by all reduce tasks=313657
Total megabyte-seconds taken by all map tasks=
Total megabyte-seconds taken by all reduce tasks=
Map-Reduce Framework
Map input records=51444
Map output records=154332
Map output bytes=2583585
Map output materialized bytes=2892267
Input split bytes=300
Combine input records=0
Combine output records=0
Reduce input groups=51444
Reduce shuffle bytes=2892267
Reduce input records=154332
Reduce output records=51444
Spilled Records=308664
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=9078
CPU time spent (ms)=1054730
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=696040
File Output Format Counters
Bytes Written=861195
16/07/17 21:04:19 INFO client.RMProxy: Connecting to ResourceManager at hadoop22/10.187.84.51:8032
16/07/17 21:04:19 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/07/17 21:04:20 INFO input.FileInputFormat: Total input paths to process : 1
16/07/17 21:04:20 INFO mapreduce.JobSubmitter: number of splits:1
16/07/17 21:04:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_5_0022
16/07/17 21:04:20 INFO impl.YarnClientImpl: Submitted application application_5_0022
16/07/17 21:04:20 INFO mapreduce.Job: The url to track the job: http://hadoop22:8088/proxy/application_5_0022/
16/07/17 21:04:20 INFO mapreduce.Job: Running job: job_5_0022
16/07/17 21:04:27 INFO mapreduce.Job: Job job_5_0022 running in uber mode : false
16/07/17 21:04:27 INFO mapreduce.Job:
map 0% reduce 0%
16/07/17 21:04:33 INFO mapreduce.Job:
map 100% reduce 0%
16/07/17 21:04:38 INFO mapreduce.Job:
map 100% reduce 100%
16/07/17 21:04:38 INFO mapreduce.Job: Job job_5_0022 completed successfully
16/07/17 21:04:38 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=24
FILE: Number of bytes written=186173
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=861298
HDFS: Number of bytes written=12
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3580
Total time spent by all reduces in occupied slots (ms)=3393
Total time spent by all map tasks (ms)=3580
Total time spent by all reduce tasks (ms)=3393
Total vcore-seconds taken by all map tasks=3580
Total vcore-seconds taken by all reduce tasks=3393
Total megabyte-seconds taken by all map tasks=3665920
Total megabyte-seconds taken by all reduce tasks=3474432
Map-Reduce Framework
Map input records=51444
Map output records=1
Map output bytes=16
Map output materialized bytes=24
Input split bytes=103
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=24
Reduce input records=1
Reduce output records=1
Spilled Records=2
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=89
CPU time spent (ms)=2360
Physical memory (bytes) snapshot=
Virtual memory (bytes) snapshot=
Total committed heap usage (bytes)=
Shuffle Errors
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=861195
File Output Format Counters
Bytes Written=12
精确度:51444
CPU time spent (ms)=1054730
(此时看来数据量很小的时候,不太适合分而治之,间接说明了hadoop适合大数据)
map tasks=3
总结:MapReduce在处理大数据的时候,会逐渐发挥集群的优势,通过mapper任务的并行处理,提高大数据的处理速度!
&&相关文章推荐
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
访问:425921次
积分:7213
积分:7213
排名:第3099名
原创:312篇
转载:63篇
评论:67条
阅读:25076
文章:117篇
阅读:110231
文章:28篇
阅读:39391
文章:22篇
阅读:21974
阅读:5583}

我要回帖

更多关于 分治算法几个经典例子 的文章

更多推荐

版权声明:文章内容来源于网络,版权归原作者所有,如有侵权请点击这里与我们联系,我们将及时删除。

点击添加站长微信