首页 技术 正文
技术 2022年11月23日
0 收藏 364 点赞 4,617 浏览 2088 个字

本文是在集群已经搭建好的基础上来说的,还没有搭建好集群的小伙伴还请自行百度!

启动spark-shell之前要先启动hive metastore 和 hiveservice2

hive --service metastore &
hiveserver2

然后再启动spark-shell

spark-shell --master yarn --deploy-medo client

启动之后可能会抛出一些异常

[root@master hadoop]# spark-shell --master yarn --deploy-mode client
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/06/04 09:46:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/06/04 09:47:00 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/06/04 09:47:35 WARN DFSClient: Caught exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:609)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.endBlock(DFSOutputStream.java:370)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:546)
18/06/04 09:47:59 WARN HiveConf: HiveConf of name hive.metastore.hbase.aggregate.stats.false.positive.probability does not exist
18/06/04 09:47:59 WARN HiveConf: HiveConf of name hive.llap.io.orc.time.counters does not exist
18/06/04 09:47:59 WARN HiveConf: HiveConf of name hive.orc.splits.ms.footer.cache.ppd.enabled does not exist
18/06/04 09:47:59 WARN HiveConf: HiveConf of name hive.server2.metrics.enabled does not exist
18/06/04 09:47:59 WARN HiveConf: HiveConf of name hive.llap.am.liveness.connection.timeout.ms does not exist
18/06/04 09:47:59 WARN HiveConf: HiveConf of name hive.server2.thrift.client.connect.retry.limit does not exist
18/06/04 09:47:59 WARN HiveConf: HiveConf of name hive.llap.io.allocator.direct does not exist
18/06/04 09:47:59 WARN HiveConf: HiveConf of name hive.llap.auto.enforce.stats does not exist
18/06/04 09:47:59 WARN HiveConf: HiveConf of name hive.llap.client.consistent.splits does not exist

这些警告不影响咱们的运行

scala> val rdd=sc.parallelize(1 to 100,5)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[] at parallelize at <console>:24scala> rdd.count
res0: Long = 100 scala>

spark的UI页面

spark-shell操作hive

相关推荐
python开发_常用的python模块及安装方法
adodb:我们领导推荐的数据库连接组件bsddb3:BerkeleyDB的连接组件Cheetah-1.0:我比较喜欢这个版本的cheeta…
日期:2022-11-24 点赞:878 阅读:9,254
Educational Codeforces Round 11 C. Hard Process 二分
C. Hard Process题目连接:http://www.codeforces.com/contest/660/problem/CDes…
日期:2022-11-24 点赞:807 阅读:5,692
下载Ubuntn 17.04 内核源代码
zengkefu@server1:/usr/src$ uname -aLinux server1 4.10.0-19-generic #21…
日期:2022-11-24 点赞:569 阅读:6,531
可用Active Desktop Calendar V7.86 注册码序列号
可用Active Desktop Calendar V7.86 注册码序列号Name: www.greendown.cn Code: &nb…
日期:2022-11-24 点赞:733 阅读:6,302
Android调用系统相机、自定义相机、处理大图片
Android调用系统相机和自定义相机实例本博文主要是介绍了android上使用相机进行拍照并显示的两种方式,并且由于涉及到要把拍到的照片显…
日期:2022-11-24 点赞:512 阅读:7,942
Struts的使用
一、Struts2的获取  Struts的官方网站为:http://struts.apache.org/  下载完Struts2的jar包,…
日期:2022-11-24 点赞:671 阅读:5,102