Skip to content

Output structure

Jump to bottom
Kenji Yip Tong edited this page Aug 25, 2023 · 24 revisions

The full output of ARPEGGIO is quite a maze, so here you can find a "map" to guide you to what you're looking for. This section goes through each one of the folders in alphabetical order.

Output overview

output
|-- benchmark
|-- Bismark
|        |-- Deduplication
|        |-- Extraction
|-- Bisulfite_Genome
|-- DMR_analysis
|        |-- Context_separation
|        |-- dmrseq
|                |-- CG_context
|                |-- CHG_context
|                |-- CHH_context
|-- FastQC
|-- FASTQtrimmed
|-- MultiQC
|        |-- multiqc_data
|        |-- multiqc_report.html
|-- qualimap
|-- read_sorting

benchmark

output
|-- benchmark

This folder contains one file per analysis (rule). Each file contains runtime information for the specific rule. For more details see here.

Bismark

Main folder

output
|-- Bismark

NOTE: All the reports in this folder are already included in the MultiQC report

The main folder includes one folder per alignment, one deduplication folder (see below) and one extraction folder (see below). Each folder with a different alignment has the following name format: <SAMPLE_NAME>_<GENOME_ALIGNED>. So if a folder is called allopolyploid123_2 it means it contains the alignment of the sample allopolyploid123 to the genome of the second parental species. Each folder includes the following files:

  • 1.<SAMPLE_NAME>_R1_val_1_bismark_bt2_PE_report.txt or 2.<SAMPLE_NAME>_R1_val_1_bismark_bt2_PE_report.txt

Report about several alignment statistics.

  • 1.<SAMPLE_NAME>_R1_val_1_bismark_bt2_pe.bam or 2.<SAMPLE_NAME>_R1_val_1_bismark_bt2_pe.bam

File with all the alignment information for all reads in bam format.

Deduplication

output
`-- Bismark
    `-- deduplication

NOTE: All the reports in this folder are already included in the MultiQC report

This folder includes one folder per deduplication with the same name format as the main folder. Each folder includes the following files:

  • 1.<SAMPLE_NAME>_R1_val_1_bismark_bt2_pe.deduplication_report.txt or 2.<SAMPLE_NAME>_R1_val_1_bismark_bt2_pe.deduplication_report.txt

Report about deduplication statistics.

  • 1.<SAMPLE_NAME>_R1_val_1_bismark_bt2_pe.deduplicated.bam or 2.<SAMPLE_NAME>_R1_val_1_bismark_bt2_pe.deduplicated.bam

File with all duplicate reads removed.

  • 1.<SAMPLE_NAME>_R1_val_1_bismark_bt2_pe.deduplicated_sorted.bam or 2.<SAMPLE_NAME>_R1_val_1_bismark_bt2_pe.deduplicated_sorted.bam

File with all duplicate reads removed and sorted by position.
Note: If the samples go through read sorting (allopolyploids with RUN_READ_SORTING=True) these files will not be present. Instead, you will have sorted bam files in your read_sorting output directory (See below). These files are generated because sorted bam files are required by Qualimap to produce mapping quality reports for MultiQC.

Extraction

output
`-- Bismark
    `-- extraction

NOTE: All the reports in this folder are already included in the MultiQC report

This folder includes one folder per extraction with a similar name format as the main folder. The only differences are: 1) the folder for parent species ends with 1 or 2 depending on the parent and 2) the folder for allopolyploids ending with 12 include merged information between folders 1 and 2. The reason of this merge is to simplify steps downstream. The prefixes of the files in these folders varies depending on which steps were run upstream and the type of data (SE or PE). Each folder (besides the folders ending in 12) have the following files:

  • <VARYING_PREFIX>.deduplicated.M-bias.txt

File with information about methylation bias (M-bias) in the reads.

  • <VARYING_PREFIX>.deduplicated.bedGraph.gz

File with cytosine methylation percentage in bedGraph format.

  • <VARYING_PREFIX>.deduplicated.bismark.cov.gz

File with cytosine methylation percentage and amount of methylated/unmethylated reads in cov format.

  • <VARYING_PREFIX>.deduplicated_splitting_report.txt

Report about methylation extraction statistics.

  • CpG_<VARYING_PREFIX>.deduplicated.txt
  • CHG_<VARYING_PREFIX>.deduplicated.txt
  • CHH_<VARYING_PREFIX>.deduplicated.txt

Context specific methylation information. More info about those files can be found here (Appendix III: "Bismark Methylation Extractor", paragraph OUTPUT).

  • <VARYING_PREFIX>.CX_report.txt

This file has information about methylation status of all cytosines in the species' genome. You can find more details about the content of the file here (Section 3, paragraph III: "Bismark methylation extractor").

Bisulfite_Genome

output
`-- Bisulfite_Genome

This file has all the information about virtual bisulfite conversion of the species' genomes.

DMR_analysis

Context_separation

output
`-- DMR_analysis
    `-- context_separation

This folder has three subfolders: parent1, parent2, allopolyploid. Each folder includes the following files:

  • <SAMPLE_NAME>_CG.cov
  • <SAMPLE_NAME>_CHG.cov
  • <SAMPLE_NAME>_CHH.cov

Each one of these files has context specific information about all cytosines having at least 1 read covering it. The format of this file is the standard cov, the columns are as follows:

chromosome      start      end        methylation %         methylated reads          unmethylated reads

dmrseq

output
`-- DMR_analysis
    `-- dmrseq

This folder has three context-specific subfolders: CG_context, CHG_context and CHH_context. The content of each one of the folders changes according to the mode ARPEGGIO was run: either standard or special.
For standard mode (allopolyploid vs diploid) the output includes the following:

  • parent1_v_allo.txt and parent2_v_allo.txt
  • parent1_v_allo_sig.bed and parent2_v_allo_sig.bed
  • parent1_v_allo_genes_overlap.txt and parent1_v_allo_genes_overlap.txt
  • DM_genes_parent1_v_allo_<CG, CHG or CHH>_context.txt and DM_genes_parent1_v_allo_<CG, CHG or CHH>_context.txt

The first file parent1_v_allo.txt is the output from dmrseq and includes statistical results for all candidate regions for a given context. You can find more information about the content of this file here. The file parent1_v_allo_sig.bed is a BED file with the genomic ranges of significant Differentially Methylated Regions (DMRs) selected from the previous file (qvalue < 0.05). The file parent1_v_allo_genes_overlap.txt shows the genic regions overlapping with DMRs from the previous BED file together with the original DMRs ranges. The last file DM_genes_parent1_v_allo_<CG, CHG or CHH>_context.txt is a summary of all genes overlapping with a DMR and has the following structure:

geneID      seqname      start      end      region_start    region_end      overlap_length    methylation_status

geneID = gene ID extracted from annotation file, seqname = chromosome/scaffold, start = start position of the gene, end = end position of the gene, region_start = start position of the DMR, region_end = end position of the DMR, overlap_length = length of the overlap between DMR and gene region, methylation_status = direction of change in methylation: increase means the allopolyploid has increased methylation, decrease means the allopolyploid has decreased methylation.

For special mode (allopolyploid vs allopolyploid or diploid vs diploid) the output includes the following:

  • A_v_B_diploid.txt or A_v_B_polyploid.txt
  • A_v_B_diploid_sig_sorted.bed or A_v_B_polyploid_sig_sorted.bed
  • A_v_B_diploid_genes_overlap.txt or A_v_B_polyploid_genes_overlap.txt
  • DM_genes_A_v_B_diploid_<CG, CHG or CHH>_context.txt or DM_genes_A_v_B_polyploid_<CG, CHG or CHH>_context.txt

The first file A_v_B_diploid.txt (or A_v_B_polyploid.txt) is the output from dmrseq and includes statistical results for all candidate regions for a given context. You can find more information about the content of this file here. The file A_v_B_diploid_sig_sorted.bed (or A_v_B_polyploid_sig_sorted.bed) is a BED file with the genomic ranges of significant Differentially Methylated Regions (DMRs) selected from the previous file (qvalue < 0.05). The file A_v_B_diploid_genes_overlap.txt (or A_v_B_polyploid_genes_overlap.txt) shows the genic regions overlapping with DMRs from the previous BED file together with the original DMRs ranges. The last file DM_genes_A_v_B_diploid_<CG, CHG or CHH>_context.txt (or DM_genes_A_v_B_polyploid_<CG, CHG or CHH>_context.txt) is a summary of all genes overlapping with a DMR and has the following structure:

geneID      seqname      start      end      region_start    region_end      overlap_length    methylation_status

geneID = gene ID extracted from annotation file, seqname = chromosome/scaffold, start = start position of the gene, end = end position of the gene, region_start = start position of the DMR, region_end = end position of the DMR, overlap_length = length of the overlap between DMR and gene region, methylation_status = direction of change in methylation: increase means that group A has increased methylation, decrease means that group A has decreased methylation.

FastQC

output
`-- FastQC

NOTE: All the reports in this folder are already included in the MultiQC report

This folder contains the output from FastQC with files named <SAMPLE_NAME>_R1_fastqc.zip and <SAMPLE_NAME>_R1_fastqc.html. The first one is a zipped folder with all the information needed for the second file: a report with all quality criteria tested for a given sample. More information about this can be found on the FastQC manual.

FASTQtrimmed

output
`-- FASTQtrimmed

NOTE: All the reports in this folder are already included in the MultiQC report

This folder contains the trimming output together with quality checks. For each sample there will be three files (name depends on Paired-End or Single-End data):

  • <SAMPLE_NAME>_R1.fastq.gz_trimming_report.txt or <SAMPLE_NAME>.fastq.gz_trimming_report.txt
  • <SAMPLE_NAME>_R1_val_1.fq.gz or <SAMPLE_NAME>_trimmed.fq.gz
  • <SAMPLE_NAME>_R1_val_1_fastqc.html or <SAMPLE_NAME>_trimmed_fastqc.html

First file has a trimming report, second file is the trimmed fastq files and third file is the quality check report for the trimmed data.

MultiQC

output
`-- MultiQC
|   |-- multiqc_data
|   `-- multiqc_report.html

MultiQC combines together reports from different tools (Bismark, FastQC, Trim Galore!, qualimap) in a single interactive report in HTML format. This report is called multiqc_report.html. The folder multiqc_report_data contains all the data included in the report.

qualimap

output
`-- qualimap

NOTE: All the reports in this folder are already included in the MultiQC report

Qualimap provides a report on alignment files with a lot of information. The Qualimap folder will include folders named <SAMPLE_NAME>_p1 or <SAMPLE_NAME>_p2 for alignment files coming from parental data and folders named <SAMPLE_NAME>_allo_<pe or se>_1 or <SAMPLE_NAME>_allo_<pe or se>_2 for alignment files coming from allopolyploid data. Each one of those sample folders will include an HTML report called qualimapReport.html with many different statistics. Other folders and files found in sample folders have information included in the report.

read_sorting

output
`-- read_sorting

This folder has one subfolder for each allopolyploid sample included in the analyses. The content of these folders is the following:

  • <SAMPLE_NAME>_classified1.ref.bam: reads in bam1 (reads mapped to ASSEMBLY_PARENT_1 in config.yaml) classified as coming from ASSEMBLY_PARENT_1
  • <SAMPLE_NAME>_classified1.alt.bam: reads in bam1 (reads mapped to ASSEMBLY_PARENT_1 in config.yaml) classified as coming from ASSEMBLY_PARENT_2
  • <SAMPLE_NAME>_classified2.ref.bam: reads in bam2 (reads mapped to ASSEMBLY_PARENT_2 in config.yaml) classified as coming from ASSEMBLY_PARENT_2
  • <SAMPLE_NAME>_classified2.alt.bam: reads in bam2 (reads mapped to ASSEMBLY_PARENT_2 in config.yaml) classified as coming from ASSEMBLY_PARENT_1
  • Both <SAMPLE_NAME>_classified1.unk.bam and <SAMPLE_NAME>_classified2.unk.bam contain reads that could not be classified, aligned to ASSEMBLY_PARENT_1 and ASSEMBLY_PARENT_2 respectively.
  • Both <SAMPLE_NAME>_classified1.mul.bam and <SAMPLE_NAME>_classified2.mul.bam, aligned to ASSEMBLY_PARENT_1 and ASSEMBLY_PARENT_2 respectively.
  • '<SAMPLE_NAME>_classified_reads.list', A tab-delimited text file with one row per read (or read pair if PE) and multiple columns holding information about mapping and classification.

For more information about these files, please visit the EAGLE-RC repository.

Next section

  1. Snakefile (rules and relationships)

Wiki index

Basic steps to run & get an idea of the workflow:

  1. Overview
  2. Input files
  3. Configuration file
  4. Running Snakemake
  5. Output structure

Advanced information to better understand & modify the workflow:

  1. Snakefile (rules and relationships)
  2. Scripts
  3. Software and tools

Clone this wiki locally