Monday, October 13, 2014

Sqoop 1.99.3 w/ Hadoop 2 Installation / Getting Started Craziness (addtowar.sh not found, common.loader, etc.)


We have a ton of data in relational databases that we are looking to migrate onto our Big Data platform. S We took an initial look around and decided Sqoop might be worth a try.   I ran into some trouble getting Sqoop up and running.  Here in lies that story...

The main problem is the documentation (and google).  It appears as though Sqoop changed install processes between minor dot releases.  Google will likely land you on this documentation:
http://sqoop.apache.org/docs/1.99.1/Installation.html

That documentation mentions a shell script, ./bin/addtowar.sh.  That shell script no longer exists in sqoop version 1.99.3.  Instead you should reference this documentation:
http://sqoop.apache.org/docs/1.99.3/Installation.html

In that documentation, they mention the common.loader property in server/conf/catalina.properties.   If you haven't been following the Tomcat scene, that is the new property that allows you to load jar files onto your classpath without dropping them into $TOMCAT/lib, or your war file. (yuck)

To get Sqoop running, you'll need all of the Hadoop jar files (and the transitive dependencies) on the CLASSPATH when Sqoop/Tomcat starts up.  And unless, you add all of the Hadoop jar files to this property, you will end up with any or all of the following CNFE/NCDFE exceptions in your log file (found in server/logs/localhost*.log):

java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobClient
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration

Through trial and error, I found all of the paths needed for the common.loader property.  I ended up with the following in my catalina.properties:

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/Users/bone/tools/hadoop/share/hadoop/common/*.jar,/Users/bone/tools/hadoop/share/hadoop/yarn/lib/*.jar,/Users/bone/tools/hadoop/share/hadoop/mapreduce/*.jar,/Users/bone/tools/hadoop/share/hadoop/tools/lib/*.jar,/Users/bone/tools/hadoop/share/hadoop/common/lib/*.jar

That got me past all of the classpath issues.  Note, in my case /Users/bone/tools/hadoop was a complete install of Hadoop 2.4.0.

I also ran into this exception:

Caused by: org.apache.sqoop.common.SqoopException: MAPREDUCE_0002:Failure on submission engine initialization - Invalid Hadoop configuration directory (not a directory or permission issues): /etc/hadoop/conf/

That path has to point to your Hadoop conf directory.   You can find this setting in server/conf/sqoop.properties.  I updated mine to:
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/Users/bone/tools/hadoop/etc/hadoop
(Again, /Users/bone/tools/hadoop is the directory of my hadoop installation)

OK ---  Now, you should be good to go!

Start the server with:
bin/sqoop.sh server start

Then, the client should work! (as shown below)

bin/sqoop.sh client
...
sqoop:000> set server --host localhost --port 12000 --webapp sqoop
Server is set successfully
sqoop:000> show version --all
client version:
  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
server version:
  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b

...

From there, follow this:
http://sqoop.apache.org/docs/1.99.3/Sqoop5MinutesDemo.html

Happy sqooping all.

29 comments:

Arjun kumar said...

Thank you so much sharing this incredible content. Your unique article helps me to update myself with the current technology. Keep up this by sharing more articles. This might help many like me to seek the best Big Data Training Chennai

Suranka VMware said...

Nice article i was really impressed by seeing this article, it was very interesting and it is very useful for Learners..
VMWare course chennai | VMWare certification in chennai | VMWare certification chennai

dhanalakshmi palu said...

very nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information.
Phonegap Training in chennai | Phonegap Training chennai | Phonegap course in chennai | Phonegap course chennai

dhanamlakshmi palu said...


Your posts is really helpful for me.Thanks for your wonderful post. I am very happy to read your post.very nice !!! AWS course chennai | AWS Certification in chennai | AWS Certification chennai

Manish Agarwal said...

Thanks a lot. This is very helpful

dhanamlakshmi palu said...

This is extremely helpful info!! Very good work. Everything is very interesting to learn and easy to understood. Thank you for giving information. cloud computing training in chennai | cloud computing training chennai | cloud computing course in chennai | cloud computing course chennai

jack wilson said...

I get a lot of great information from this blog. Thank you for your sharing this informative blog. Just now I have completed hadoop certification course at a leading academy. If you are interested to learn Hadoop Training in Chennai visit FITA IT training and placement academy.

john son said...

Hi friends,This is Johnson from Chennai.Thanks for sharing this informative blog. I did Unix certification course in Chennai at Fita academy. This is really useful for me to make a bright career.
Regards..
Unix Training Institutes in Chennai

kovalan Jayamurugan said...

Hi admin thanks for sharing informative article on hadoop technology. In coming years, hadoop and big data handling is going to be future of computing world. This field offer huge career prospects for talented professionals. Thus, taking Hadoop Training in Chennai will help you to enter big data technology.

Roshini RS said...

This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.
Regards,
Informatica training in chennai|Best Informatica Training In Chennai|Informatica training chennai

Jannik Andrew said...

The Hadoop tutorial you have explained is most useful for begineers who are taking Hadoop Administrator Online Training for Installing Haddop on their Own
Thank you for sharing Such a good tutorials on Hadoop

Sai Santosh said...

Great information shared on this website. Thanks a lot and keep up the good work and research on hadoop. Regards, hadoop online training

Simran Kaur said...

Great information about Hadoop. It will be helpful for us.
Thank you !!
I would like to share useful things for hadoop job seekers Hadoop Interview Questions & Answers.

shankar said...



Great information, thansk for this.Its very informative and i am sure it will help many other people like the way it helps me. Thanks for the information.


Software Estimation Techniques Training in Chennai

Abdurrahman Adebiyi said...

Call me crazy... but the only way I could get sqoop2 going on was copying all the Hadoop jar files to the $SQOOP2_HOME/server/lib.

For some reason I just couldn't get the "common.loader" property in the catalina.properties file to work.

Abdurrahman Adebiyi said...

By the way I built version 1.99.6 of sqoop2

Tek Classes said...

Well defined thanks for sharing

Visit - www.tekclasses.in
for more information about Big data.

Jhon Abraham said...

Testing is the best in Information Technolgohy compared to anyother domain. Thanks for sharing your knowledge via this content. You have presented it very well. Thank you once again. I have bookmarked this page for future use. Keep blogging article like this.

Selenium training in chennai | Selenium training chennai | Best selenium training in chennai

Mervin Parmar said...

Using big data analytics may give the companies many fruitful results, the findings can be implemented in their business decisions so as to minimize their risk and to cut the costs.
hadoop training in chennai|big data training|big data training in chennai

Raghu Sharma said...

Cloud computing is the next big thing, through cloud the users have the liberty to use a shared network. The companies can focus on core business parts rather than investing heavily on infrastucture.
cloud computing training in chennai|cloud computing courses in chennai|cloud computing training

mack alex said...

Great post
It is very informative thanks for sharing

Hadoop training in bangalore

Specto IT Training said...

This is a perfect post. I like your writing style. Will look around your blog.

oracle Apps SCM R12 Finance Functional online training

big data trunk1 said...




Thank you for sharing such great information !

big data classroom training

hadoop ónlinÉ training

free big data bootcamp

hadoop big data videos

Big data QA Tester training

Big data Analyst training

jhansi joe said...

Thanks for the valid post, it's crystal clear step. The expansion of internet and other business intelligence leads to large volume of data. Industries are looking for talented professionals to maintain and process huge volume of data with latest tools available in the market. Taking Hadoop Training in Chennai | Big Data Training in Chennai will ensure better career prospects for talented professionals.

Devi.Angularjs said...

Interesting Article

Angularjs Training Institutes in Chennai | Backbone.JS Training Institutes in Chennai | Bootstrap Training Institutes in Chennai | Node.js Training Institutes in Chennai | Typescript Training Institutes in Chennai

Devi.Angularjs said...

JavaScript Training in Chennai | Angular 2 Training in Chennai | HTML5 Training in Chennai | JQuery Training in Chennai

Devi.Angularjs said...

JavaScript Training in Chennai | Angular 2 Training in Chennai | HTML5 Training in Chennai | JQuery Training in Chennai

punyavathisiva said...

This blog is gives great information on big data hadoop online training in hyderabad, uk, usa, canada.

best online hadoop training in hyderabad.
hadoop online training in usa, uk, canada.

jazz said...

I just see the post i am so happy to the communication science post of information's.So I have really enjoyed and reading your blogs for these posts.Any way I’ll be replay for your great thinks and I hope you post again soon...
Software Testing Training in Chennai