How do I add a new column to a Spark DataFrame (using PySpark)?
I have a Spark DataFrame (using PySpark 1.5.1) and would like to add a new column.
apache-spark-sql
‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
I’m trying to load an SVM file and convert it to a DataFrame so I can use the ML module (Pipeline ML) from Spark.
I’ve just installed a fresh Spark 1.5.0 on an Ubuntu 14.04 (no spark-env.sh configured).
Encode and assemble multiple features in PySpark
I have a Python class that I’m using to load and process some data in Spark. Among various things I need to do, I’m generating a list of dummy variables derived from various columns in a Spark dataframe. My problem is that I’m not sure how to properly define a User Defined Function to accomplish what I need.
Count number of non-NaN entries in each column of Spark dataframe with Pyspark
I have a very large dataset that is loaded in Hive. It consists of about 1.9 million rows and 1450 columns. I need to determine the “coverage” of each of the columns, meaning, the fraction of rows that have non-NaN values for each column.
Does spark predicate pushdown work with JDBC?
According to this
Explode in PySpark
I would like to transform from a DataFrame that contains lists of words into a DataFrame with each word in its own row.
Retrieve top n in each group of a DataFrame in pyspark
There’s a DataFrame in pyspark with data as below:
How to export a table dataframe in PySpark to csv?
I am using Spark 1.3.1 (PySpark) and I have generated a table using a SQL query. I now have an object that is a DataFrame. I want to export this DataFrame object (I have called it “table”) to a csv file so I can manipulate it and plot the columns. How do I export the DataFrame “table” to a csv file?
How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
We are using the PySpark libraries interfacing with Spark 1.3.1.
Pyspark: explode json in column to multiple columns
The data looks like this –