Thursday, September 25, 2014

[YARN] : NullPointerException in MRClientService.getHttpPort()

Have you moved your Hadoop jobs over to Yarn?  Are you seeing the following NullPointerException coming out of your job?
Caused by: java.lang.NullPointerException
 at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:167)
 at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:150)
 ... 14 more
This appears to be a poorly handled case in Hadoop, where the webApp fails to load.  You need to look at the full log file.  You will likely see the following in the log preceeding the NPE:
ERROR (org.apache.hadoop.mapreduce.v2.app.client.MRClientService:139) - Webapps failed to start. Ignoring for now:
That line is produced by the following code in YARN:
    LOG.info("Instantiated MRClientService at " + this.bindAddress);
    try {
      webApp = WebApps.$for("mapreduce", AppContext.class, appContext, "ws").with(conf).
          start(new AMWebApp());
    } catch (Exception e) {
      LOG.error("Webapps failed to start. Ignoring for now:", e);
    }
    super.start();
This means your webApp failed to load.  In my case, this was caused by dependency conflicts on my classpath.  (servlet & jasper compiler)  I've also seen it cause by a dependency mssing, such as:
java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer
In my situation, the solution was to cleanup my classpath, but the webApp might fail for lots of reasons, which will result in an NPE later in the log file when MRClientService attempts to getHttpPort without a webApp, which is shown in the following code:
public int getHttpPort() {
    return webApp.port();
  }
Anyway, that is probably more detail than you wanted. Happy yarning.

6 comments:

Shash said...

Could you please mention the procedure you took for the classpath clean-up? I am facing the same issue. Thanks in advance.

Paul Miller said...

This is one such interesting and useful article that i have ever read. The way you have structured the content is so realistic and meaningful. Thank you so much for sharing this in here. Keep up this good work and I'm expecting more contents like this from you in future.

Big Data Training in Chennai | Big Data Training | Hadoop Course in Chennai

AadvikKumar said...



Nice post, I bookmark your blog because I found very good information on your blog, Thanks for sharing more informatiom



Android App Development

Lenova Services said...

Thanks for sharing this valuable information.
lenovo thinkpad service center chennai
lenovo laptop service center in chennai
lenovo ideapad service center chennai

1croreprojects said...

be projects in chennai

ns2 projects in chennai

ieee java projects in chennai

ieee dotnet projects in chennai

bsc projects in chennai

msc projects in chennai

bca projects in chennai

mba projects in chennai

Iqbal Dlk said...

very useful blogs.
hotels near us consulate chennai
hotels near apollo hospital chennai
hotels near uk embassy chennai
hotels near german consulate chennai
hotels near sankara nethralaya chennai
business class hotels in chennai