org.apache.spark.sql.hive.test

Class TestHiveContext

Object

org.apache.spark.sql.SQLContext

org.apache.spark.sql.hive.HiveContext

org.apache.spark.sql.hive.test.TestHiveContext

All Implemented Interfaces:

java.io.Serializable, Logging, CacheManager, SQLConf, UDFRegistration

Direct Known Subclasses:

TestHive

public class TestHiveContext
extends HiveContext

A locally running test instance of Spark's Hive execution engine.

Data from testTables will be automatically loaded whenever a query is run over those tables. Calling reset will delete all tables and other state in the database, leaving the database in a "clean" state.

TestHive is singleton object version of this class because instantiating multiple copies of the hive metastore seems to lead to weird non-deterministic failures. Therefore, the execution of test cases that rely on TestHive must be serialized.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	TestHiveContext.QueryExecution Override QueryExecution with special debug workflow.
class	TestHiveContext.TestTable

Nested classes/interfaces inherited from interface org.apache.spark.sql.SQLConf

SQLConf.Deprecated$

Constructor Summary

Constructors

Constructor and Description
TestHiveContext(SparkContext sc)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	cacheTables()
scala.util.matching.Regex	describedTable()
TestHiveContext.QueryExecution	executePlan(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)
java.io.File	getHiveFile(String path)
scala.Option<java.io.File>	hiveDevHome() The location of the hive source code.
java.io.File	hiveFilesTemp()
scala.Option<java.io.File>	hiveHome() The location of the compiled hive distribution
scala.collection.Seq<TestHiveContext.TestTable>	hiveQTestUtilTables()
java.io.File	inRepoTests()
void	loadTestTable(String name)
String	metastorePath()
int	numShufflePartitions() Fewer partitions to speed up testing.
scala.collection.mutable.HashMap<String,TestHiveContext.TestTable>	registerTestTable(TestHiveContext.TestTable testTable)
void	reset() Resets the test instance by deleting any tables that have been created.
scala.collection.Seq<String>	runSqlHive(String sql) Runs the specified SQL query using Hive.
scala.collection.mutable.HashMap<String,TestHiveContext.TestTable>	testTables() A list of test tables and the DDL required to initialize them.
java.io.File	testTempDir()
String	warehousePath()

Methods inherited from class org.apache.spark.sql.hive.HiveContext

analyze, convertMetastoreParquet, createTable, dialect, hivePlanner, hiveql, hql, setConf, sql

Methods inherited from class org.apache.spark.sql.SQLContext

applySchema, applySchemaToPythonRDD, applySchemaToPythonRDD, baseRelationToSchemaRDD, createParquetFile, createSchemaRDD, dropTempTable, extraStrategies, jsonFile, jsonFile, jsonFile, jsonRDD, jsonRDD, jsonRDD, logicalPlanToSparkQuery, parquetFile, parseDataType, registerRDDAsTable, sparkContext, table

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Methods inherited from interface org.apache.spark.sql.SQLConf

autoBroadcastJoinThreshold, clear, codegenEnabled, columnBatchSize, columnNameOfCorruptRecord, defaultSizeInBytes, externalSortEnabled, getAllConfs, getConf, getConf, inMemoryPartitionPruning, isParquetBinaryAsString, parquetCompressionCodec, parquetFilterPushDown, setConf, settings, useCompression

Methods inherited from interface org.apache.spark.sql.CacheManager

cachedData, cacheLock, cacheQuery, cacheTable, clearCache, invalidateCache, isCached, lookupCachedData, lookupCachedData, readLock, tryUncacheQuery, uncacheQuery, uncacheTable, useCachedData, writeLock

Methods inherited from interface org.apache.spark.sql.UDFRegistration

registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerPython

Constructor Detail

TestHiveContext

public TestHiveContext(SparkContext sc)

Method Detail

warehousePath

public String warehousePath()

metastorePath

public String metastorePath()

testTempDir

public java.io.File testTempDir()

hiveHome

public scala.Option<java.io.File> hiveHome()

The location of the compiled hive distribution

hiveDevHome

public scala.Option<java.io.File> hiveDevHome()

The location of the hive source code.

runSqlHive

public scala.collection.Seq<String> runSqlHive(String sql)

Description copied from class: HiveContext

Runs the specified SQL query using Hive.

executePlan

public TestHiveContext.QueryExecution executePlan(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)

numShufflePartitions

public int numShufflePartitions()

Fewer partitions to speed up testing.

hiveFilesTemp

public java.io.File hiveFilesTemp()

inRepoTests

public java.io.File inRepoTests()

getHiveFile

public java.io.File getHiveFile(String path)

describedTable

public scala.util.matching.Regex describedTable()

testTables

public scala.collection.mutable.HashMap<String,TestHiveContext.TestTable> testTables()

A list of test tables and the DDL required to initialize them. A test table is loaded on demand when a query are run against it.

registerTestTable

public scala.collection.mutable.HashMap<String,TestHiveContext.TestTable> registerTestTable(TestHiveContext.TestTable testTable)

hiveQTestUtilTables

public scala.collection.Seq<TestHiveContext.TestTable> hiveQTestUtilTables()

cacheTables

public boolean cacheTables()

loadTestTable

public void loadTestTable(String name)

reset

public void reset()

Resets the test instance by deleting any tables that have been created. TODO: also clear out UDFs, views, etc.