Launching spark-shell on a DCOS master node

Important note: The following guide is obsolete, it won't work with modern versions of DC/OS Spark packages. Please use Apache Zeppelin instead.

  1. SSH to the DCOS master node.
  2. Download spark-shell: wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz
  3. Unpack it: tar -xzf spark-1.5.1-bin-hadoop2.6.tgz 
  4. Change the working directory to the spark binary folder: cd spark-1.5.1-bin-hadoop2.6/bin/
  5. Start spark-shell: ./spark-shell --master mesos://<local-ip>:5050, where <local-ip> is a local ip of your master node (you can see it in your shell prompt or launching the ifconfig command).

Note: This approach was only smoke-tested and probably should be extended. spark-shell establishes a connection with a Spark master and you can launch some Spark commands, but there are still some warnings during the shell initialization, for example, one regarding a lack of SQL connection.

Have more questions? Submit a request

Comments

Powered by Zendesk