Notice
Recent Posts
Recent Comments
Link
일 | 월 | 화 | 수 | 목 | 금 | 토 |
---|---|---|---|---|---|---|
1 | 2 | |||||
3 | 4 | 5 | 6 | 7 | 8 | 9 |
10 | 11 | 12 | 13 | 14 | 15 | 16 |
17 | 18 | 19 | 20 | 21 | 22 | 23 |
24 | 25 | 26 | 27 | 28 | 29 | 30 |
Tags
- C
- 하둡2
- effective python
- 주식분석
- hadoop2
- hive
- LSTM
- tensorflow
- 선형대수
- graph
- python
- GRU
- HelloWorld
- Java
- 텐서플로
- 알고리즘
- NumPy
- Sort
- yarn
- 딥러닝
- 코딩더매트릭스
- 파이썬
- scrapy
- codingthematrix
- 하이브
- collections
- C언어
- 그래프이론
- RNN
- recursion
Archives
- Today
- Total
EXCELSIOR
매퍼(Mapper) 와 리듀서(Reducer) 클래스 본문
1. Mapper.java
public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> { //<입력키 유형, 입력값 유형, 출력키 유형, 출력값 유형>을 의미 public class Context extends MapContext<KEYIN,VALUEIN,KEYOUT,VALUEOUT> { public Context(Configuration conf, TaskAttemptID taskid, RecordReader<KEYIN,VALUEIN> reader, RecordWriter<KEYOUT,VALUEOUT> writer, OutputCommitter committer, StatusReporter reporter, InputSplit split) throws IOException, InterruptedException { super(conf, taskid, reader, writer, committer, reporter, split); } } /** * Called once at the beginning of the task. */ protected void setup(Context context ) throws IOException, InterruptedException { // NOTHING } /** * Called once for each key/value pair in the input split. Most applications * should override this, but the default is the identity function. */ @SuppressWarnings("unchecked") protected void map(KEYIN key, VALUEIN value, Context context) throws IOException, InterruptedException { //맵 메서드, 맵리듀스 프로그램을 개발할 때 이 메서드를 재정의한다. context.write((KEYOUT) key, (VALUEOUT) value); } /** * Called once at the end of the task. */ protected void cleanup(Context context ) throws IOException, InterruptedException { // NOTHING } /** * Expert users can override this method for more complete control over the * execution of the Mapper. * @param context * @throws IOException */ public void run(Context context) throws IOException, InterruptedException { setup(context); while (context.nextKeyValue()) { map(context.getCurrentKey(), context.getCurrentValue(), context); } cleanup(context); } }
2. Reducer.java
public class Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT> { public class Context extends ReduceContext<KEYIN,VALUEIN,KEYOUT,VALUEOUT> { public Context(Configuration conf, TaskAttemptID taskid, RawKeyValueIterator input, Counter inputKeyCounter, Counter inputValueCounter, RecordWriter<KEYOUT,VALUEOUT> output, OutputCommitter committer, StatusReporter reporter, RawComparator<KEYIN> comparator, Class<KEYIN> keyClass, Class<VALUEIN> valueClass ) throws IOException, InterruptedException { super(conf, taskid, input, inputKeyCounter, inputValueCounter, output, committer, reporter, comparator, keyClass, valueClass); } } /** * Called once at the start of the task. */ protected void setup(Context context ) throws IOException, InterruptedException { // NOTHING } /** * This method is called once for each key. Most applications will define * their reduce class by overriding this method. The default implementation * is an identity function. */ @SuppressWarnings("unchecked") protected void reduce(KEYIN key, Iterable<VALUEIN> values, Context context //이 메서드를 재정의한다. ) throws IOException, InterruptedException { for(VALUEIN value: values) { context.write((KEYOUT) key, (VALUEOUT) value); } } /** * Called once at the end of the task. */ protected void cleanup(Context context ) throws IOException, InterruptedException { // NOTHING } /** * Advanced application writers can use the * {@link #run(org.apache.hadoop.mapreduce.Reducer.Context)} method to * control how the reduce task works. */ public void run(Context context) throws IOException, InterruptedException { setup(context); while (context.nextKey()) { context.progress(); reduce(context.getCurrentKey(), context.getValues(), context); } cleanup(context); } }
'DataBase > Hadoop' 카테고리의 다른 글
하둡2 예제실행 (0) | 2016.11.11 |
---|---|
하둡2 설치 및 실행 (가상 분산 모드) (4) | 2016.11.08 |
하둡2 - 네임노드 HA (0) | 2016.11.07 |
하둡2 - 얀(YARN) (0) | 2016.11.04 |
MapReduce 개념 (0) | 2016.10.13 |
Comments