Hadoop Sequence Parser (HSP)

Hadoop Sequence Parser (HSP) is a Java library that allows to parse DNA sequence reads from FASTQ/FASTA datasets stored in the Hadoop Distributed File System (HDFS).

HSP supports the processing of input datasets compressed with Gzip (i.e., .gz extension) and BZip2 (i.e., .bz2 extension) codecs. However, when considering compressed data that will be later processed by Hadoop or any other data processing engine (e.g., Spark), it is important to understand whether the underlying compression format supports splitting, as many codecs need the whole input stream to uncompress successfully. On the one hand, Gzip does not support splitting and HSP will not split the gzipped input dataset. This will work, but probably at the expense of lower performance. On the other hand, BZip2 does compression on blocks of data and later these blocks can be decompressed independent of each other (i.e., it supports splitting). Therefore, BZip2 is the recommended codec to use with HSP for better performance. This does not apply to paired-end input: the splits of a BZip2 file are ranges of compressed bytes, and the two files of a pair compress to different sizes, so the same range would not hold the same reads on both sides. HSP reads each BZip2 file of a pair in a single split (like Gzip) and logs a warning; for better performance with paired-end data, use uncompressed files.

Getting Started

Prerequisites

• Make sure you have Java Development Kit (JDK) version 11 or above

• Make sure you have a working Apache Maven distribution version 3 or above

  • See https://maven.apache.org/install.html

• HSP is built against Apache Hadoop 3.3 (which is a provided dependency, so the Hadoop version is the one of your cluster or engine). Use HSP 1.3 for Java 8 or earlier Hadoop versions

Installation

In order to download, compile, build and install the HSP library in your Maven local repository (by default ~/.m2), just execute the following commands:

git clone https://github.com/rreye/hsp.git
cd hsp
mvn install

Usage

In order to use the HSP library in your projects, add the following dependency section to your pom.xml:

<dependencies>
...
  <dependency>
   <groupId>es.udc.gac</groupId>
   <artifactId>hadoop-sequence-parser</artifactId>
   <version>2.0.0</version> <!-- or latest version -->
  </dependency>
...
</dependencies>

For single-end datasets, HSP generates <key,value> pairs of type <LongWritable, Text>. The key is the byte offset in file for each read and the value is the text-based content of the read (e.g., read name, bases and qualities for FASTQ). The Text object representing the sequence can be converted to a String object using the static method getRead() provided by the SingleEndSequenceRecordReader class. The only exception to the key being a byte offset in file is BZip2 input: the keys of the splits after the first one are not file offsets, as they combine the compressed start of the split with the uncompressed bytes read from it.

For paired-end datasets, HSP generates <key,value> pairs of type <LongWritable, PairText>. The key is the byte offset in file for each paired read and the value is a tuple containing the pair of the Text objects that represent the paired sequence. HSP provides static methods in the PairedEndSequenceRecordReader class that allow obtaining "left" and "right" reads separately as String objects: getLeftRead() and getRightRead(), respectively.

Configuration

HSP is configured through the Hadoop configuration of the job (Configuration in Hadoop, the config object in the Spark examples, the Job configuration in Flink), either setting the property or with the static methods of es.udc.gac.hadoop.sequence.parser.util.Configuration:

Property	Default	Static method	Description
hsp.input.buffer.size	65536	setInputBufferSize(conf, bytes)	Size in bytes of the input buffer of the readers (also the initial size of the Text that holds each record)
hsp.trim.sequence.name	false	setTrimSequenceName(conf, true)	Cut the read name at its first space, dropping the description that follows it (e.g. @ERR188245.1 length=75 becomes @ERR188245.1). By default the name is kept as it is in the file. Enabling it makes the records shorter, which saves memory

An invalid value (a buffer size that is not a positive integer, a flag other than true or false) makes the reader fail with an IllegalArgumentException that names the property.

Logging

HSP logs through the SLF4J API and does not bundle any logging backend: messages go wherever your application, Hadoop, Spark or Flink send their logs. The logger names are the HSP class names (package es.udc.gac.hadoop.sequence.parser). At DEBUG level each input split is described when it is opened and closed, and a WARN is emitted if a FASTQ file ends in the middle of a record. Nothing is logged per record.

Upgrading from HSP 1.3

The full list of changes is in the CHANGELOG. These are the ones that may require action in applications that use HSP:

• Read names are no longer trimmed by default. HSP 1.3 cut the read name at its first space; HSP 2.0 keeps the name as it is in the file. To get the previous behaviour call Configuration.setTrimSequenceName(conf, true) (or set hsp.trim.sequence.name to true).
• Java 11 and Hadoop 3.3 are required. Keep using HSP 1.3 for Java 8 or earlier Hadoop versions.
• Logging backend: HSP no longer bundles Log4j2 and only uses the SLF4J API, so the logging configuration is the one of your application or engine.
• Invalid configuration values (a buffer size that is not a positive integer, a flag other than true or false) now fail with an IllegalArgumentException instead of being ignored.
• Paired-end properties: the keys stored by setLeftInputPath() and setRightInputPath() are now hsp.paired.* instead of hsra.paired.*. Nothing changes if you only use the static methods.
• BZip2 paired-end input is read in a single split per file (see above).

Hadoop examples

Setting up the input format of a Hadoop job for a single-end dataset in FASTQ format stored in "/path/to/file":

Job job = Job.getInstance(new Configuration(), "Example");
Path inputFile = new Path("/path/to/file");

//Set input path and input format class for HSP
SingleEndSequenceInputFormat.addInputPath(job, inputFile);
job.setInputFormatClass(FastQInputFormat.class);

Setting up the input format of a Hadoop job for a paired-end dataset in FASTQ format stored in "/path/to/file" and "/path/to/file2":

Job job = Job.getInstance(new Configuration(), "Example");
Path inputFile1 = new Path("/path/to/file1");
Path inputFile2 = new Path("/path/to/file2");

//Set input format class and input paths for HSP
job.setInputFormatClass(PairedEndSequenceInputFormat.class);
PairedEndSequenceInputFormat.setLeftInputPath(job, inputFile1, FastQInputFormat.class);
PairedEndSequenceInputFormat.setRightInputPath(job, inputFile2, FastQInputFormat.class);

Spark examples

Creating a Spark RDD from a single-end dataset in FASTQ format stored in "/path/to/file" using Java:

SparkSession sparkSession = SparkSession.builder().config(new SparkConf()).getOrCreate();
JavaSparkContext jsc = JavaSparkContext.fromSparkContext(sparkSession.sparkContext());
Configuration config = jsc.hadoopConfiguration();

// Create RDD
JavaPairRDD<LongWritable, Text> readsRDD = jsc.newAPIHadoopFile("/path/to/file", FastQInputFormat.class, LongWritable.class, Text.class, config);

Creating a Spark RDD from a paired-end dataset in FASTQ format stored in "/path/to/file1" and "/path/to/file2" using Java:

SparkSession sparkSession = SparkSession.builder().config(new SparkConf()).getOrCreate();
JavaSparkContext jsc = JavaSparkContext.fromSparkContext(sparkSession.sparkContext());
Configuration config = jsc.hadoopConfiguration();

// Set left and right input paths for HSP
PairedEndSequenceInputFormat.setLeftInputPath(config, "/path/to/file1", FastQInputFormat.class);
PairedEndSequenceInputFormat.setRightInputPath(config, "/path/to/file2", FastQInputFormat.class);

// Create RDD
JavaPairRDD<LongWritable, PairText> readsRDD = jsc.newAPIHadoopFile("path/to/file1", PairedEndSequenceInputFormat.class, LongWritable.class, PairText.class, config);

Flink examples

Creating a Flink DataSet from a single-end dataset in FASTQ format stored in "/path/to/file" using Java:

ExecutionEnvironment flinkExecEnv = ExecutionEnvironment.getExecutionEnvironment();
Job hadoopJob = Job.getInstance();

// Set input path for HSP
SingleEndSequenceInputFormat.setInputPaths(hadoopJob, "/path/to/file");

// Create DataSet
DataSet<Tuple2<LongWritable,Text>> readsDS = flinkExecEnv.createInput(new HadoopInputFormat<LongWritable,Text>(FastQInputFormat.class, LongWritable.class, Text.class, hadoopJob));

Creating a Flink DataSet from a paired-end dataset in FASTQ format stored in "/path/to/file1" and "/path/to/file2" using Java:

ExecutionEnvironment flinkExecEnv = ExecutionEnvironment.getExecutionEnvironment();
Job hadoopJob = Job.getInstance();
Configuration config = hadoopJob.getConfiguration();

// Set left and right input paths for HSP
PairedEndSequenceInputFormat.setLeftInputPath(config, "/path/to/file1", FastQInputFormat.class);
PairedEndSequenceInputFormat.setRightInputPath(config, "/path/to/file2", FastQInputFormat.class);

// Create DataSet
DataSet<Tuple2<LongWritable, PairText>> readsDS = flinkExecEnv.createInput(new HadoopInputFormat<LongWritable, PairText>(new PairedEndSequenceInputFormat(), LongWritable.class, PairText.class, hadoopJob));

Projects using HSP

• MarDRe
• HSRA
• SeQual
• SMusket
• SparkEC

Authors

HSP is developed in the Computer Architecture Group at the Universidade da Coruña by:

• Roberto R. Expósito (http://gac.udc.es/~rober)
• Luis Lorenzo Mosquera (https://github.com/luislorenzom)
• Jorge González-Domínguez (http://gac.udc.es/~jgonzalezd)

License

This library is distributed as free software and is publicly available under the GPLv3 license (see the LICENSE file for more details)