Repository files navigation

Authors: Sebastian Mackowiak & Marc Friedländer

This is miRDeep2 developed by Sebastian Mackowiak & Marc Friedländer.miRDeep2 discovers active known or novel miRNAs from deep sequencing data(Solexa/Illumina, 454, ...).

(minor edits toREADME,TUTORIAL,CHANGELOG, andFAQ, convertion toMarkdown, trailing whitespace removal & CI setup by Marcel Schilling)

Linux system, 2GB Ram, enough disk space dependent on your deep sequencing data

MacOSX with Xcode and gcc compiler installed. (This can be obtained from theappstore, if there are any issues with installing it please look for helponline).

To compile the Vienna package it may be necessary to have GNU grep installedsince the MacOSX grep is BSD based and sometimes not accepted by the installer.To get a GNU grep you could for example install homebrew by typing

ruby -e"$(curl -fsSL \  https://raw.githubusercontent.com/Homebrew/install/master/install)"

(the link could be out of date, in that case look up online what to do)

After that typing

brew tap homebrew/dupes; brew install grep

will install GNU grep asggrep in/usr/local/bin/

Type

perl install.pl

Follow the instructions given below

First download all necessary packages listed here

1. bowtie short read aligner
2. Vienna package with RNAfold
3. SQUID library goto Squid and download it
4. randfold
5. Perl package PDF::API2

When packages are downloaded

1. attach the miRDeep2 executable path to your PATH

echo'export PATH=$PATH:your_path_to_mirdeep2/src'>>~/.bashrc

2. unzip bowtie-0.11.3-bin-linux-x86_64.zip

3. put the bowtie directory into yourPATH variable,e.g.

echo'export PATH=$PATH:your_path_tobowtie'>>~/.bashrc

4. tar xvvzf ViennaRNA-1.8.4.tar.gz

5. cd to the Vienna dir

6. type

./configure --prefix=your_path_to_Vienna/install_dirmakemake install

7. add Vienna binaries to yourPATH variable,e.g.

echo'export PATH=$PATH:your_path_to_Vienna/install_dir/bin'>>~/.bashrc

8. tar xxvzf squid-1.9g.tar.gz

9. tar xvvzf randfold-2.0.tar.gz

10. cd randfold2.0

11. edit Makefile,e.g.emacs Makefile:

change line withINCLUDE=-I. toINCLUDE=-I. -I<your_path_to_squid-1.9g> -L<your_path_to_squid-1.9g>,e.g.INCLUDE=-I. -I/home/Pattern/squid-1.9g/ -L/home/Pattern/squid-1.9g/

12. make

13. add randfold to yourPATH variable,e.g.

echo'export PATH=$PATH:your_path_to_randfold'>>~/.bashrc

14. tar xvvzf PDF-API2-0.73.tar.gz

15. cd to your PDF_API2 directory

16. then type in

perl Makefile.PL INSTALL_BASE=your_path_to_miRDeep2 LIB=your_path_to_miRDeep2/libmakemaketestmake install

17. add your library to thePERL5LIB,e.g.

echo \'export PERL5LIB=PERL5LIB:your_path_to_miRDeep2/lib/perl5' \>>~/.bashrc

18. cd to your mirdeep2 directory (the one containinginstall.pl)

19. touch install_successful

20. start a new shell session to apply the changes to environment variables

To test if everything is installed properly type in

1. bowtie
2. RNAfold -h
3. randfold
4. make_html.pl

You should not get any error messages. Otherwise something is not correctlyinstalled.

Everything that is download by the installer will be in a directory called<your_path_to_mirdeep2>/essentials

miRDeep2 analyses can be performed using the three scriptsmiRDeep2.pl,mapper.pl andquantifier.pl.

Wrapper function for themiRDeep2.pl program package. The script runs allnecessary scripts of the miRDeep2 package to perform a microRNA detection deepsequencing data anlysis.

• A FASTA file with deep sequencing reads,
• a FASTA file of the corresponding genome,
• a file of mapped reads to the genome in miRDeep2 ARF format,
• an optional FASTA file with known miRNAs of the analysed species, and
• an optional FASTA file of known miRNAs of related species.

• A spreadsheet and
• an HTML file

with an overview of all detected miRNAs in the deep sequencing input data.

The miRDeep2 module identifies known and novel miRNAs in deep sequencing data.The output of the mapper module can be directly plugged into the miRDeep2module.

The user wishes to identify miRNAs in mouse deep sequencing data, using defaultoptions.ThemiRBase_mmu_v14.fa file contains all miRBase mature mouse miRNAs, whilethemiRBase_rno_v14.fa file contains all the miRBase mature rat miRNAs.The2> will pipe all progress output to thereport.log file.

miRDeep2.pl reads_collapsed.fa genome.fa reads_collapsed_vs_genome.arf \  miRBase_mmu_v14.fa miRBase_rno_v14.fa precursors_ref_this_species.fa \  -t Mouse2>report.log

This command will generate

• a directory with PDFs showing the structures, read signatures and scorebreakdowns of novel and known miRNAs in the data,
• an HTML webpage that links to all results generated (result.html),
• a copy of the novel and known miRNAs contained in the webpage but in textformat which allows easy parsing (result.csv),
• a copy of the performance survey contained in the webpage but in text format(survey.csv), and
• a copy of the miRNA read signatures contained in the PDFs but in text format(output.mrd).

The user wishes to identify miRNAs in deep sequencing data from an animal withno related species in miRBase:

miRDeep2.pl reads_collapsed.fa genome.fa reads_collapsed_vs_genome.arf \  none none none2>report.log

This command will generate the same type of files as example use 1 above.Note that there it will in practice always improve miRDeep2 performance ifmiRNAs from some related species is input, even if it is not closely related.

Processes reads and/or maps them to the reference genome.

Default input is

• a file in FASTA,seq.txt orqseq.txt format.

More input can be given depending on the options used.

The output depends on the options used (see below).

Either

• a FASTA file with processed reads, or
• an ARF file with with mapped reads, or
• both

are output.

The mapper module is designed as a tool to process deep sequencing reads and/ormap them to the reference genome. The module works in sequence space, and canprocess or map data that is in sequence FASTA format.A number of the functions of the mapper module are implemented specificallywith Solexa/Illumina data in mind. For example on how to post-process mappingsin color space, see example use 5:

The user wishes to parse a file inqseq.txt format to FASTA format, convertfrom RNA to DNA alphabet, remove entries with non-canonical letters (lettersother thana,c,g,t,u,n,A,C,G,T,U, orN), clipadapters, discard reads shorter than 18 nts and collapse the reads:

mapper.pl reads_qseq.txt -b -h -i -j -k TCGTATGCCGTCTTCTGCTTGT -l 18 -m \ -s reads_collapsed.fa

The user wishes to map a FASTA file against the reference genome.The genome has already been indexed bybowtie-build.The first of the indexed files is namedgenome.1.ebwt:

mapper.pl reads_collapsed.fa -c -p genome -t reads_collapsed_vs_genome.arf

The user wishes to process the reads as in example use 1 and map the reads asin example use 2 in a single step, while observing the progress:

mapper.pl reads_qseq.txt -b -h -i -j -k TCGTATGCCGTCTTCTGCTTGT -l 18 -m \  -p genome -s reads_collapsed.fa -t reads_collapsed_vs_genome.arf -v

The user wishes to parse a GEO file to FASTA format and process it as inexample use 1.The GEO file is in tabular format, with the first column showing the sequenceand the second column showing the read counts:

geo2fasta.pl GSM.txt> reads.famapper.pl reads.fa -c -h -i -j -k TCGTATGCCGTCTTCTGCTTGT -l 18 -m \  -s reads_collapsed.fa

The user has already removed 3' adapters in color space and has mapped thereads against the genome using the BWA tool. The BWA output file is namedreads_vs_genome.sam. Notice that the BWA output contains extra fields thatare not required for SAM format. Our converter requires these fields and thusmay not work with all types of SAM files. The user wishes to generatereads_collapsed.fa andreads_vs_genome.arf to input to miRDeep2:

bwa_sam_converter.pl reads_vs_genome.sam reads.fa reads_vs_genome.arfmapper.pl reads.fa -c -i -j -l 18 -m -s reads_collapsed.fa

The module maps the deep sequencing reads to predefined miRNA precursors anddetermines by that the expression of the corresponding miRNAs.First, the predefined mature miRNA sequences are mapped to the predefinedprecursors. Optionally, predefined star sequences can be mapped to theprecursors too. By that the mature and star sequence in the precursors aredetermined.Second, the deep sequencing reads are mapped to the precursors. The number ofreads falling into an interval 2 nt upstream and 5 nt downstream of themature/star sequence is determined.

• A FASTA file with precursor sequences,
• a FASTA file with mature miRNA sequences,
• a FASTA file with deep sequencing reads, and
• optionally a FASTA file with star sequences and the 3 letter code of thespecies of interest.

• A 2 column table file calledmiRNA_expressed.csv with miRNA identifiers andits read count,
• a file calledmiRNA_not_expressed.csv with all miRNAs having 0 read counts,
• a signature file calledmiRBase.mrd,
• a file calledexpression.html that gives an overview of all miRNAs theinput data, and
• a directory calledpdfs that contains for each miRNA a PDF file showing itssignature and structure.

quantifier.pl -p precursors.fa -m mature.fa -r reads.fa

It creates a file calledresult.html that gives an overview of miRDeep2detected miRNAs (known and novel ones). The HTML file lists up each detectedmiRNA and provides among others information on its miRDeep2 score, reads mappedto its mature, loop and star sequence, the mature, star and consensus precursorsequences themselves and provides links to BLAST, BLAT, mirBase for miRBasemiRNAs and to a PDF file that shows the signature and structure.

• A miRDeep2 output.mrd file and
• a miRDeep2 survey.csv file

• Aresult.html file with an entry for each provisional miRNA that containsinformation about its assigned Id, miRDeep2 score, estimated probability thatthe miRNA candidate is a true positive, rfam alert, total read count, matureread count, loop read count, star read count, significant randfold p-value,miRBase miRNA, example miRBase miRNA with the same seed, BLAT, BLAST,consensus mature sequence, consensus star sequence and consensus precursorsequence. Furthermore, the miRBase miRNAs existent in the input data but notscored by miRDeep2 are listed.
• A directory calledpdfs that contains for each provisional miRNA ID a PDFwith its signature and structure.
• A file calledresult.csv (when option-c is used) that contains the sameentries as the HTML file.

make_html.pl -f miRDeep_outfile -s survey.csv -c -e -y 123456789

Removes 3' end adaptors from deep sequenced small RNAs. The script searches foroccurrences of the six first nucleotides of the adapter in the read sequence,starting after position 18 in the read sequence (so the shortest clipped readwill be 18 nts). If no matches to the first six nts of the adapter areidentified in a read, the 3' end of the read is searched for shorter matches tothe 5 to 1 first nts of the adapter.

• A FASTA file with the deep sequencing reads and the adapter sequence (both inRNA or DNA alphabet).

• A FASTA file with the clipped reads.

FASTA IDs are retained. If no matches to the adapter prefixes are identified ina given read, the unclipped read is output.

clip_adapters.pl reads.fa TCGTATGCCGTCTTCTGCTTGT> reads_clipped.fa

It is possible to clip adapters using more sophisticated methods.Users are encouraged to test other methods with the miRDeep2 modules.

Collapses reads in the FASTA file to ensure that each sequence only occursonce.To indicate how many times reads the sequence represents, a suffix is added toeach FASTA identifier.E.g. a sequence that represents ten reads in the datawill have the_x10 suffix added to the identifier.

• A FASTA file, either in standard format or in the collapsed suffix format.

• A FASTA file in the collapsed suffix format.

collapse_reads.pl reads.fa> reads_collapsed

Since the script reads all FASTA entries into a hash using the sequence as key,it can potentially use more than 3 GB memory when collapsing very big datasets,>50 million reads. In this case, the user can partition the reads(for instance based on the 5' nucleotide), collapse separately and concatenate.

This script is a wrapper forexcise_precursors.pl, which it calls one or moretimes, incrementing the height of the read stack required for initiatingexcision until the number of excised precursors falls below a given threshold.

• The reference genome in FASTA format,
• the mapped reads in.arf format,
• a filename that the excised precursors will be written to, and
• the maximal number of precursors that should be reported.

excise_precursors_iterative.pl genome.fa reads_vs_genome.arf \  potential_precursors.fa 50000 -a

Excises precursors from the genome using the mapped reads as guidelines.

• The reference genome in FASTA format and
• the mapped reads in.arf format.

• The excised precursors in FASTA format.

excise_precursors.pl genome.arf reads_vs_genome.arf -b

Performs simple filtering of entries in a FASTA file.

• A FASTA file.

• A filtered FASTA file.

fastaparse.pl reads.fa -a 18 -s> reads_no_short.fa

This script only prints out the FASTA entries that match an ID in the ID file.

• A FASTA file and a file with IDs, one ID per line.

• A FASTA file containing the FASTA entries that match an ID.

fastaselect.pl reads.fa reads_select.ids> reads_select.fa

Scans a file searching for the suffixes that are generated bycollapse_reads.pl (e.g._x10).It sums up the integer values in the suffixes and outputs the sum. If a givenid occurs multiple times in the file, it will multi-count the integer value ofthe ID. It will also only count the first integer occurrence in a given line.

• Any file containing the suffixes that are generated bycollapse_reads.pl.

This will typically be a FASTA file or a list of IDs.

• The sum of integer values (the total read count).

find_read_count.pl reads_collapsed.fa

Parses GSM format files into FASTA format.

• GSM files in tabular format.

The first column should be sequences and the second column the number of timesthe sequence occurs in the data.

• A FASTA file, one sequence per line (the sequences are expanded).

geo2fasta.pl GSM.txt> reads.fa

Parsesseq.txt orqseq.txt output from the Solexa/Illumina platform toFASTA format.

• Aseq.txt or
• qseq.txt file.

By defaultseq.txt.

• A FASTA file, one entry for each line ofseq.txt.

The entries are namedseq plus a running number that is incremented by onefor each entry. Any. characters in theseq.txt file is substituted with anN.

illumina_to_fasta.pl s_1.qseq.txt -a> reads.fa

For each potential miRNA precursor input, the miRDeep2 core algorithm eitherdiscards it or assigns it a log-odds score that reflects the probability thatthe precursor is a genuine miRNA.

Default input is

• an ARF file with the read signatures and
• an RNAfold output file with the structures of the potential miRNA precursors.

• A .mrd file with all potential miRNA precursors that are scored.

miRDeep2_core_algorithm.pl signature.arf potential_precursors.str \  -s miRBase_related_species.fa -y potential_precursors.rand> output.mrd

The-z option has not been thoroughly tested.

Performs simple filtering of entries in an.arf file.

Default input is

• an.arf file.

• A filtered.arf file.

parse_mappings.pl reads_vs_genome.arf -a 0 -b 18 -c 25 -i 5 \> reads_vs_genome_parsed.arf

Performs a designated number of rounds of permuted controls (for details, seeFriedländer et al., Nature Biotechnology, 2008).

The permutation controls estimate the number of false positives produced by amiRDeep2_core_algorithm.pl run.The input toperform_controls.pl should be

• a file containing the exact command line used to initiate themiRDeep2_core_algorithm.pl run,
• the structure file input tomiRDeep2_core_algorithm.pl, and
• the desired rounds of controls.

• A file in.mrd format.

The output of each control run is separated by a linepermutation integer.The mean number of entries output by the control runs gives an estimate of thefalse positives produced. The further contents (besides the number of entries)of the.mrd output byperform_controls.pl is not biologically meaningful.

perform_controls.pl line potential_precursors.str 100 \> output_controls.mrd

In a file output by RNAfold, each entry can be partitioned into an 'id' partand an 'other' part, consisting of the dot-bracket structure, sequence, mfeetc. This scripts reads all 'id' parts into a hash and pairs them with 'other'parts from random entries. This is used by theperform_controls.pl script.

• An RNAfold output file.

• An RNAfold output file with IDs moved to random entries.

permute_structure.pl potential_precursors.str \> potential_precursors_permuted.str

Prepares the signature file to be input to themiRDeep2_core_algorithm.plscript.

• A FASTA file with deep sequencing reads and
• a FASTA file with precursors.

• A signature file in.arf format.

prepare_signature.pl reads_collapsed.fa potential_precursors.fa \  -a miRBase_this_species.fa> signature.arf

Substitutesus andUs toTs.This is useful sincebowtie does not matchUs toTs.

• A FASTA file.

• A substituted FASTA file.

rna2dna.pl reads_RNA_alphabet.fa> reads_DNA_alphabet.fa

This script identifies potential precursors whose structure is basicallyconsistent with Dicer recognition.Since running randfold is time-consuming, it is practical only to estimatep-values for those potential precursors that actually fold into hairpinstructures.

• An ARF file with the read signatures and
• an RNAfold output file with the structures of the potential miRNA precursors.

• A list of ids, separated by newlines.

select_for_randfold.pl signature.arf potential_precursors.str \> potential_precursors_for_randfold.ids

Surveys miRDeep2 performance at score cut-offs from -10 to 10.

Default input is

• a.mrd file output by themiRDeep2_core_algorithm.pl script.

• A .csv file with performace statistics.

survey.pl output.mrd -a output_controls.mrd -b miRBase_this_species.fa \  -c signature.arf -d 2> survey.csv

It converts abowtiebwt mapping file to amirdeeparf file.

• A file inbwt format.

• A file inmirdeeparf format.

It converts abwasam mapping file to amirdeeparf file.

• Abwa created file insam format.

• A file inmirdeeparf format.

It gives information about thebwa FLAG in abwa created mapping file insam format.

• A FLAG number created bybwa.

• Information about the alignment created bybwa.

Removes 3' end adaptors from deep sequenced small RNAs.The script searches for occurrences of the six first nucleotides of the adapterin the read sequence, starting after position 18 in the read sequence (so theshortest clipped read will be 18 nts). If no matches to the first six nts ofthe adapter are identified in a read, the 3' end of the read is searched forshorter matches to the 5 to 1 first nts of the adapter.

• A FASTA file with the deep sequencing reads and
• the adapter sequence (both in RNA or DNA alphabet).

• A FASTA file with the clipped reads.

FASTA IDs are retained. If no matches to the adapter prefixes are identified ina given read, the unclipped read is output.

clip_adapters.pl reads.fa TCGTATGCCGTCTTCTGCTTGT> reads_clipped.fa

It is possible to clip adapters using more sophisticated methods. Users areencouraged to test other methods with the miRDeep2 modules.

It checks the supplied genome FASTA file for its correctness.Identifier lines are not allowed to contain whitespaces and must be unique.Sequence lines are not allowed to contain characters others thanA,C,G,T,N,a,c,g,t, orn.

• A genome file in FASTA format

It checks the supplied mapping file for its correctness.Each line in the file must be in the ARF format.

• A mapping file in ARF format.

It checks the suppliedmature_miRNA FASTA file for its correctness.Identifier lines are not allowed to contain whitespaces and must be unique.Sequence lines are not allowed to contain characters others thanA,C,G,T,N,a,c,g,t, orn.

• A mature miRNA file in FASTA format.

It checks the supplied reads file for its correctness.Each identifier line must have the format of '>name_uniqueNumber_xnumbere.g.>xyz_1_x20. See also fileformat_descriptions.txt` for more detailedinformations.

• A mapping file in ARF format.

Extracts mature and precursor sequences from miRBase fasta files forspecies of interest.

• A fasta file from miRBAase
• One or more species three letter code abbreviations

• A fasta file in a proper format usable by quantifier.pl and miRDeep2.pl.
• Multiline sequences from input files are put on a single line and MacOS and Windows linebreaks/carriage returns are removed

extract_miRNAs.pl mature_miRBase.fa hsa> mature_hsa.faextract_miRNAs.pl hairpin_miRBase.fa hsa> hairpin_hsa.faextract_miRNAs.pl mature_miRBase.fa mmu,chi> mature_other.fa