pyspark - Spark output when submitting via Yarn cluster vs. client -


i new spark , got running on cluster (spark 2.0.1 on 9 node cluster running community version of mapr). submit wordcount example via

./bin/spark-submit --master yarn --jars ~/hadoopperma/jars/hadoop-lzo-0.4.21-snapshot.jar examples/src/main/python/wordcount.py ./readme.md

and following output

17/04/07 13:21:34 warn client: neither spark.yarn.jars nor spark.yarn.archive set, falling uploading libraries under spark_home. : 68 help: 1 when: 1 hadoop: 3 ...

looks working properly. when add --deploy-mode cluster following output:

17/04/07 13:23:52 warn client: neither spark.yarn.jars nor spark.yarn.archive set, falling uploading libraries under spark_home.

so no errors but not seeing wordcount results. missing? see job in history server , says completed successfully. checked user directory in dfs no new files written except empty directory: /user/myuser/.sparkstaging

code (wordcount.py example shipped spark):

from __future__ import print_function import sys operator import add pyspark.sql import sparksession   if __name__ == "__main__":     if len(sys.argv) != 2:         print("usage: wordcount <file>", file=sys.stderr)         exit(-1)      spark = sparksession\         .builder\         .appname("pythonwordcount")\         .getorcreate()      lines = spark.read.text(sys.argv[1]).rdd.map(lambda r: r[0])     counts = lines.flatmap(lambda x: x.split(' ')) \               .map(lambda x: (x, 1)) \               .reducebykey(add)     output = counts.collect()     (word, count) in output:         print("%s: %i" % (word, count))      spark.stop()

the reason output not printing is:

when run in spark-client mode node on initiating job driver , when collect result collected on node , print it.

in yarn-cluster mode driver other node not 1 through initiated job. when call .collect function result collected on , printed on node. can find result being printed in sys-out of driver. better approach write output somewhere in hdfs.

the reason spark.yarn.jars warning is:

in order run spark job yarn needs binaries available on nodes of cluster if these binaries not available part of job preparation, spark create zip file jars under $spark_home/jars , upload distributed cache.

to solve :

by default, spark on yarn use spark jars installed locally, spark jars can in world-readable(chmod 777) location on hdfs. allows yarn cache on nodes doesn't need distributed each time application runs. point jars on hdfs, example, set spark.yarn.jars hdfs:///some/path.

after placing jars run code :

./bin/spark-submit --master yarn --jars ~/hadoopperma/jars/hadoop-lzo-0.4.21-snapshot.jar examples/src/main/python/wordcount.py ./readme.md --conf spark.yarn.jars="hdfs:///some/path"

source : http://spark.apache.org/docs/latest/running-on-yarn.html


Comments

Post a Comment

Popular posts from this blog

How to understand 2 main() functions after using uftrace to profile the C++ program? -

i trying uftrace profile following simple c++ program: #include <iostream> class { public: a() {std::cout << "a created" << std::endl;} ~a() {std::cout << "a destroyed" << std::endl;} }; int main() { a; return 0; } the profile result this: # uftrace a.out created destroyed # duration tid function 2.026 [ 4828] | __cxa_atexit(); [ 4828] | main() { [ 4828] | __static_initialization_and_destruction_0() { 89.397 [ 4828] | std::ios_base::init::init(); 0.768 [ 4828] | __cxa_atexit(); 93.029 [ 4828] | } /* __static_initialization_and_destruction_0 */ 94.425 [ 4828] | } /* main */ [ 4828] | main() { [ 4828] | a::a() { 11.104 [ 4828] | std::operator <<(); 10.825 [ 4828] | std::basic_ostream::operator <<(); 24.514 [ 4828] | } /* a::a */ [ 4828] | a::~a() { 0.978 [ 4828] | ...
Read more

c# - Update a combobox from a presenter (MVP) -

i using mvp in project, new in mvp. have 2 comboboxes. when select option in combobox, combobox should filled new data. this action in presenter. view 'view1' in presenter, , introduced combobox1 , combobox2 properties in 'view1', because need 'datasource', 'displaymember', 'valuemember' , 'refresh()' in method below. but, when using pattern, enough send property like public string combobox2 { { return combobox1.selectedvalue.tosstring(); } } into presenter not whole combobox. how can solve problem? public void onselectedindexchangedcombobox1() { if (view1.combobox1.selectedindex == -1) { return; } datatable dt = tools.getdatatable("a path"); var query = (from o in dt.asenumerable() o.field<string>("afield") == farmerview.combobox1.selectedvalue.tostring() orderby o.field<string>("anotherfield") select...
Read more

How to put a lock and transaction on table using spring 4 or above using jdbcTemplate and annotations like @Transactional? -

i trying put lock on table while writing on table , if happened in between roll-back . trying convert below code lock table test_g1 read; lock table test_g write; -- begin; start transaction; insert test_g1 values(143); insert test_g values(145); select * test_g1; select * test_g; rollback; select * test_g; unlock tables; how convert above code @transactional spring jdbctemplate code? @transactional(rollbackfor={dataaccessexception.class}) public void test(){ jdbctemplate.execute("insert test1 (id, nam) values (4, 'a')"); throw new dataaccessexception("error") { }; } here trying throw error, insert statement should rollback it's not happening . thanks edit-1 attaching code , doing in jdbcdaoimpl.java , mentioned problem comment above test() . app.java package com.cgiri.javabrains.spring4; import org.springframework.beans.factory.annotation.autowired; import org.springframework.context.applic...
Read more