Null-Pointer Assignment Partitioning

you should be able to use show table extended partition to see if you can get info on it and not try to open anyone who is zero bytes. Like this:

scala> var sqlCmd="show table extended from mydb like 'mytable' partition (date_time_date='2017-01-01')"
sqlCmd: String = show table extended from mydb like 'mytable' partition (date_time_date='2017-01-01')

scala> var partitionsList=sqlContext.sql(sqlCmd).collectAsList
partitionsList: java.util.List[org.apache.spark.sql.Row] =
[[mydb,mytable,false,Partition Values: [date_time_date=2017-01-01]
Location: hdfs://mycluster/apps/hive/warehouse/mydb.db/mytable/date_time_date=2017-01-01
Serde Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
Storage Properties: [serialization.format=1]
Partition Parameters: {rawDataSize=441433136, numFiles=1, transient_lastDdlTime=1513597358, totalSize=4897483, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}, numRows=37825}
]]

Let me know if that works and you can avoid the 0 byter's with such or if you still get null pointer..

James

This can happen in the scenario where table definition and specific partition definition is different, and the underlying data matches table definition but not partition definition.

When a table with partitions is altered to add a column using statement:

ALTER TABLE <tableName> ADD COLUMNS (c1 int);

The table definition for existing partitions don't get modified as per the above statement. As a result of this there is a mismatch between partition and table definition. 

This is ok if the partition data matches the definition of partition, but if the data matches definition of table itself, NPE is thrown as there is a mismatch in data vs definition.

To avoid this issue, this statement should be used in hadoop2

ALTER TABLE <tableName> ADD COLUMNS (c1 int) CASCADE;

In case of hadoop1, CASCADE option is not available. Hence, as long as the table is external table, following can be done:

  1. Drop and recreate partitions for this table
  2. Alter partition definition for specific partition having issues
Was this article helpful?0 out of 0 found this helpful

Have more questions? Submit a request

Comments

One thought on “Null-Pointer Assignment Partitioning

Leave a Reply

Your email address will not be published. Required fields are marked *