Specify the text/string type as the value of the Hadoop counter
The current method of setting/incrementing the Hadoop counter only accepts long values.
For example, increment(long incr) and setValue(long value) are two methods I extracted from Hadoop Javadocs.
My requirement is to store more complex types of information as part of the counter (as key/value pairs). This information may involve (string, string) key-value pairs.
How do I achieve this using Hadoop counters?
If this is not possible, are there any other data structures/facilities in Hadoop/MR that allow such miscellaneous information to be stored, which can later be retrieved by specifying job_id, etc.
Thank you
Parameters
Solution
The counter is valid because the count is the sum of the counts. Each task has its own counters that can be aggregated upwards. The string does not exactly have the same type of information (how do I increment a string?). )。
Check out ZooKeeper for this. It is ideal for storing miscellaneous information and coordinating between processes. You can create a znode (maybe job_id?) that represents the job run? ), and then take a separate string as a child node.