Skip to content

Running Hive Implementation

jpbarany edited this page Jun 24, 2014 · 4 revisions

This algorithm is implemented using HIVE with Hadoop streaming. The python file AggregateMicroPath.py generates and executes HQL to create data tables and execute queries. The scripts folder contains additional python files that are used with HADOOP streaming to process the data.

Running

Running an AMP job (assuming the config and data are in place) only requires a single python command however it is best to write a shell script in order to perform any desired setup/cleanup as well as data exporting. An example can be found in run_ais.sh.

The command to run AMP is:

python AggregateMicroPath.py -c config.ini

This assumes you are currently in the hive-streaming directory and that config.ini is your configuration file. When completed, the count results will be found in a Hive table called "micro_path_intersect_counts_" with your trip_name appended (e.g. micro_path_intersect_counts_ais_small_final). Similar tables can be found for velocity and direction results.

Paris Example

Clone this wiki locally