Singles with python transformation: “cannot recognize input near ‘transform'” error
I have a Hive table that tracks the state of objects as they move through various stages of the process. The table is as follows:
hive> desc journeys;
object_id string
journey_statuses array<string>
Here is a typical logging example:
12345678 ["A","A","A","B","B","B","C","C","C","C","D"]
The
records in the table are generated using collect_list
of Hive 0.13, and the states are ordered (I’ll use collect_set
if the order isn’t important object_id).
I wrote a quick Python script that reads from standard input:
#!/usr/bin/env python
import sys
import itertools
for line in sys.stdin:
inputList = eval(line.strip())
readahead = iter(inputList)
next(readahead)
result = []
for id, (a, b) in enumerate(itertools.izip(inputList, readahead)):
if id == 0:
result.append(a)
if a != b:
result.append(b)
print result
I plan to use it in the Hive transform
call. It seems to work when running locally :
$ echo '["A","A","A","B","B","B","C","C","C","C","D"]' | python abbreviate_list.py
['A', 'B', 'C', 'D']
However, when I add a file and try to execute in Hive, an error is returned:
hive> add file abbreviateList.py;
Added resource: abbreviateList.py
hive> select
> object_id,
> transform(journey_statuses) using 'python abbreviateList.py' as journey_statuses_abbreviated
> from journeys;
NoViableAltException( ... wall of Java error messages ... )
FAILED: ParseException line 3:2 cannot recognize input near 'transform' '(' 'journey_statuses' in select expression
Can you see what I’m doing wrong?
Solution
Obviously, you can’t select other fields that aren’t in the transformation (object_id in your example). This other SO question seems to indirectly solve this problem:
How can select a column and do a TRANSFORM in Hive?
Theoretically, you can modify Python to accept object_id as an input parameter and make it a passthrough to another output field when you need to include it in the output.