Merge pull request #3106 from antgonza/2021.05

2021.03->2021.05

Author: mestaki

CHANGELOG.md

@@ -1,5 +1,13 @@
 # Qiita changelog
 
+Version 2021.05
+---------------
+
+* Replaced vis.js for cytoscape.js to display the processing networks.
+* The commands available to users, originally only filtered by input type, are now also limited by the preparation type. The options are taken from the [recommended workflows](https://qiita.ucsd.edu/workflows/).
+* Added a new [spades](https://github.com/ablab/spades) assembly pipeline for "Genome Isolate".
+* Fixed the following issues: [#3070](https://github.com/qiita-spots/qiita/issues/3070), [#3089](https://github.com/qiita-spots/qiita/issues/3089), [#2968](https://github.com/qiita-spots/qiita/issues/2968), [#3102](https://github.com/qiita-spots/qiita/issues/3102), and [#3079](https://github.com/qiita-spots/qiita/issues/3079).
+
 
 Version 2021.03
 ---------------

README.rst

@@ -3,7 +3,7 @@ Qiita (canonically pronounced *cheetah*)
 
 |Build Status| |Coverage Status|
 
-Advances in sequencing, proteomics, transcriptomics and metabolomics are giving
+Advances in sequencing, proteomics, transcriptomics, metabolomics, and others are giving
 us new insights into the microbial world and dramatically improving our ability
 to understand their community composition and function at high resolution.
 These new technologies are generating vast amounts of data, even from a single
@@ -17,16 +17,16 @@ compute resources to the global community, alleviating the technical burdens,
 such as familiarity with the command line or access to compute power, that are
 typically limiting for researchers studying microbial ecology.
 
-Qiita is currently in beta status. We are very open to community
+Qiita is currently in production/stable status. We are very open to community
 contributions and feedback. If you're interested in contributing to Qiita,
 see `CONTRIBUTING.md <https://github.com/qiita-spots/qiita/blob/master/CONTRIBUTING.md>`__.
 If you'd like to report bugs or request features, you can do that in the
 `Qiita issue tracker <https://github.com/qiita-spots/qiita/issues>`__.
 
 To install and configure your own Qiita server, see
-`INSTALL.md <https://github.com/qiita-spots/qiita/blob/master/INSTALL.md>`__.
+`INSTALL.md <https://github.com/qiita-spots/qiita/blob/master/INSTALL.md>`__. However, Qiita is not designed to be used locally but rather on a server, we therefore advise against installing your own version on a personal computer. Nevertheless, it can run just fine on a laptop or small computer for development and educational purposes. For example, for every single PR and release, we install Qiita from scratch as GitHub Actions, you can follow `these steps <https://github.com/qiita-spots/qiita/actions>`__.
 
-For more specific details about qiita visit `the Qiita main site tutorial <https://qiita.microbio.me/static/doc/html/qiita-philosophy/index.html>`__.
+For more specific details about Qiita's philosophy and design visit `the Qiita main site tutorial <https://qiita.microbio.me/static/doc/html/qiita-philosophy/index.html>`__.
 
 Current features
 ----------------
@@ -39,9 +39,9 @@ Current features
 * Easy long-term sequence data deposition to the European Nucleotide Archive (ENA),
   part of the European Bioinformatics Institute (EBI) for private and public
   studies.
-* Raw data processing for `Target Gene, Metagenomic, Metabolomic and BIOM files <https://qiita.ucsd.edu/static/doc/html/processingdata/index.html#processing-recommendations>`. BIOM files can be added as new preparation files for downstream analyses; however, this cannot be made public.
-
-* Basic downstream analyses using Qiime2.
+* Raw data processing for `Target Gene, Metagenomic, Metabolomic, Genome Isolates and BIOM files <https://qiita.ucsd.edu/static/doc/html/processingdata/index.html#processing-recommendations>`__. NOTE: BIOM files can be added as new preparation files for downstream analyses; however, this cannot be made public in the system.
+* Basic downstream analyses using QIIME 2. Note that Qiita produces qza/qzv in the analytical steps but you can also convert `non QIIME 2 artifacts <https://qiita.ucsd.edu/static/doc/html/faq.html#how-to-convert-qiita-files-to-qiime2-artifacts>`__.
+* Bulk download of `studies and artifacts <https://qiita.ucsd.edu/static/doc/html/downloading.html>`__.
 * Basic study search in the study listing page.
 * Complex metadata search via redbiom.
 
@@ -55,20 +55,7 @@ Accepted raw files
 * Multiplexed FASTQ: forward, reverse (optional), and barcodes
 * Per sample FASTQ: forward and reverse (optional)
 * Multiplexed FASTA/qual files
-
-Roadmap
--------
-
-The following is a non-exhaustive list of features that we plan to add in the
-future.
-
-* Integration of other pipelines via artifacts. Processing of raw data in
-  external sources. For example, metabolomics processing in
-  `GNPS <http://gnps.ucsd.edu>`__ and data visualization in Qiita.
-* Creation of a REST API to query and access the data hosted by Qiita.
-* Improved analysis pipeline for target gene datasets.
-* Crowd-sourced metadata curation of existing studies: improve the metadata of
-  existing studies by submitting a fix proposals to the authors of the study.
+* Per sample FASTA, only for "Full Length Operon"
 
 
 .. |Build Status| image:: https://github.com/qiita-spots/qiita/actions/workflows/qiita-ci.yml/badge.svg

qiita_core/__init__.py

@@ -6,4 +6,4 @@
 # The full license is in the file LICENSE, distributed with this software.
 # -----------------------------------------------------------------------------
 
-__version__ = "2021.03"
+__version__ = "2021.05"

qiita_db/__init__.py

@@ -27,7 +27,7 @@
 from . import user
 from . import processing_job
 
-__version__ = "2021.03"
+__version__ = "2021.05"
 
 __all__ = ["analysis", "artifact",  "archive", "base", "commands",
            "environment_manager", "exceptions", "investigation", "logger",

qiita_pet/__init__.py

@@ -6,4 +6,4 @@
 # The full license is in the file LICENSE, distributed with this software.
 # -----------------------------------------------------------------------------
 
-__version__ = "2021.03"
+__version__ = "2021.05"

qiita_pet/handlers/api_proxy/__init__.py

@@ -38,7 +38,7 @@
 from .user import (user_jobs_get_req)
 from .util import check_access, check_fp
 
-__version__ = "2021.03"
+__version__ = "2021.05"
 
 __all__ = ['prep_template_summary_get_req', 'data_types_get_req',
            'study_get_req', 'sample_template_filepaths_get_req',

qiita_pet/support_files/doc/source/processingdata/processing-recommendations.rst

@@ -6,14 +6,13 @@ Currently, Qiita supports the processing of raw data from:
 #. Target gene barcoded sequencing
 #. Shotgun sequencing
 #. Metatranscriptome sequencing
-
+#. Genome Isolate sequencing
 
 Note that the selected processing recommendations are mainly guided towards performing meta-analyses,
 this is combine different studies, even from different wet lab techniques or
 sequencing technologies. However, these parameters shouldn't prevent you using the
 resulting tables as your primary analytical source.
 
-
 Target gene barcoded sequencing
 -------------------------------
 
@@ -242,3 +241,15 @@ Additionally, a summary file showing the proportion of reads matching to each of
 Default options have been set to report only the best alignment per read reaching E-value.
 For non ribo-depleted samples (i.e. total RNA), the ribosomal reads obtained from SortMeRNA can be further used in taxonomic/compositional analysis.
 In the case of ribo-depleted samples, only the non-ribosomal reads are used in downstream analyses such as assembly, mapping, differential gene abundance analyses etc.
+
+
+Genome Isolate Processing
+-------------------------
+
+This workflow can be used for assembling (meta)-genomes (isolate and/or metagenomic data) using SPAdes v3.15.2 at set k-mer lengths of 21,33,55,77,99 and 127.
+
+The assembled contigs are stored in per sample FASTA files (originally scaffolds.fna in SPAdes).
+
+The --merge option merges the forward and reverse reads prior to assembly (preferable for isolate or metagenomes with high sequencing depth), the non-merge option works well for shallow shotgun data and/or complex environmental communities.
+
+The --meta flag is used to assemble metagenomic datasets.

qiita_ware/__init__.py

@@ -6,4 +6,4 @@
 # The full license is in the file LICENSE, distributed with this software.
 # -----------------------------------------------------------------------------
 
-__version__ = "2021.03"
+__version__ = "2021.05"

scripts/qiita-recover-jobs

@@ -70,7 +70,7 @@ def _count_jobs_in_scheduler():
     j1 = len(check_output(['qstat']).decode('ascii').split("\n"))
     # now, let's count the jobs in job arrays
     lines = check_output(['qstat', '-f']).decode('ascii').split("\n")
-    j2 = sum([int(x.split(',')[-1].split('-')[-1])
+    j2 = sum([int(x.split(' ')[-1].split(',')[-1].split('-')[-1])
               for x in lines if 'job_array_request' in x])
     return j1 + j2
 

setup.py

@@ -10,7 +10,7 @@
 from setuptools import setup
 from glob import glob
 
-__version__ = "2021.03"
+__version__ = "2021.05"
 
 
 classes = """