Oozie is a popular workflow management tool for BigData applications.
To give some high level idea, following is the container allocation for a typical oozie workflow application with hive action.
If you are running a heavy job through Oozie, there are chances that the yarn container which runs oozie job (‘Oozie workflow’ container in above image) may give out of memory.
The memory allocation for yarn container for oozie can be increased with property ‘oozie.launcher.mapreduce.map.memory.mb’ & 'oozie.launcher.mapreduce.map.java.opts'. The default value is typically 1536.
The property can be updated in oozie application workflow definition (workflow.xml) to allocate additional memory to container.
<global>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>/user/hive/conf/hive-default.xml</job-xml>
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>${metastore}</value>
</property>
<property>
<name>hive.metastore.client.socket.timeout</name>
<value>3600</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>-1</value>
</property>
<property><name>oozie.launcher.mapreduce.map.memory.mb</name>
<value>3072</value>
</property>
<property>
<name>oozie.launcher.mapreduce.map.java.opts</name>
<value>-Xmx2560m</value>
</property>
<property>
<name>oozie.launcher.yarn.app.mapreduce.am.resource.mb</name>
<value>3072</value>
</property>
<property>
<name>oozie.launcher.yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx2560m</value>
</property>
</configuration>
</global>
- Sarang
No comments:
Post a Comment