Enable file logging for Hadoop YARN Resource Manager and Node Manager

There is issue with file logging when running Hadoop yarn.cmd on Windows platform. This happens to both Hadoop 2.4.0 and 2.5.0. By default, you will not see log files for Resource Manager and Node Manager. It is caused by wrong configure of log4j.properties path in yarn.cmd. Following steps tell how to fix it (use Hadoop 2.5.0 for example, should be same for 2.4.0).

Update hadoop-2.5.0\bin\yarn.cmd: enable log4j debugging, so you can know what is happening with log4j.

set YARN_OPTS=%YARN_OPTS% -Dlog4j.debug=true

set java_arguments=%JAVA_HEAP_MAX% %YARN_OPTS% -classpath %CLASSPATH% %CLASS% %yarn-command-arguments%

Update hadoop-2.5.0\bin\yarn.cmd: remove log4j.properties in the end of "%YARN_CONF_DIR%\rm-config\log4j.properties", and move it to the beginning of classpath. The reason is that we need to specify the directory of log4j.properties file rather than its full path. Also moving it to be begining of classpath will make sure it will be loaded first.

:resourcemanager

 set CLASSPATH=%YARN_CONF_DIR%\rm-config;%CLASSPATH%

 set CLASS=org.apache.hadoop.yarn.server.resourcemanager.ResourceManager

 set YARN_OPTS=%YARN_OPTS% %YARN_RESOURCEMANAGER_OPTS%

 if defined YARN_RESOURCEMANAGER_HEAPSIZE (

    set JAVA_HEAP_MAX=-Xmx%YARN_RESOURCEMANAGER_HEAPSIZE%m

 )

 goto :eof

Create file etc\hadoop\rm-config\log4j.properties:

log4j.rootLogger=info, stdout, R

log4j.appender.stdout=org.apache.log4j.ConsoleAppender

log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

# Pattern to output the caller's file name and line number.

log4j.appender.stdout.layout.ConversionPattern=%d %5p [%t] (%F:%L) - %m%n

log4j.appender.R=org.apache.log4j.RollingFileAppender

log4j.appender.R.File=c:/app/hadoop-2.5.0/logs/ResourceManager.log

log4j.appender.R.MaxFileSize=20MB

# Keep one backup file

log4j.appender.R.MaxBackupIndex=1

log4j.appender.R.layout=org.apache.log4j.PatternLayout

log4j.appender.R.layout.ConversionPattern=%d %p %t %c - %m%n

Now launch YARN, and you will see the log files as configured in log4j.properties file.