2021.05

Author: antgonza

CHANGELOG.md

@@ -1,5 +1,26 @@
 # Qiita changelog
 
+Version 2021.05
+---------------
+
+* Replaced vis.js for cytoscape.js to display the processing networks.
+* The commands available to users, originally only filtered by input type, are now also limited by the preparation type. The options are taken from the [recommended workflows](https://qiita.ucsd.edu/workflows/).
+* Added a new [spades](https://github.com/ablab/spades) assembly pipeline for "Genome Isolate".
+* Fixed the following issues: [#3070](https://github.com/qiita-spots/qiita/issues/3070), [#3089](https://github.com/qiita-spots/qiita/issues/3089), [#2968](https://github.com/qiita-spots/qiita/issues/2968), [#3102](https://github.com/qiita-spots/qiita/issues/3102), and [#3079](https://github.com/qiita-spots/qiita/issues/3079).
+
+
+Version 2021.03
+---------------
+
+* Fixed [issue](https://github.com/qiita-spots/qtp-target-gene/issues/32) that left behind non gz per sample FASTQ files.
+* [Recommended Workflows](https://qiita-rc.ucsd.edu/workflows/) are now stored in the database.
+* Added a new button only for owners and admins within the Study page to display a list of all the Analyses generated with that study; helpful to clean up Studies and for general information.
+* The Qiita CI now runs as [GitHub Actions](https://github.com/qiita-spots/qiita/actions); moving away from Travis CI.
+* Prep information file object now stores its creation and modification timestamps.
+* Improved creation time for all information files via the to_dataframe() method.
+* Split the "other" category of the storage stats plot (https://qiita.ucsd.edu/stats/) into "other" and "biom" so biom can be its own category.
+* Added a processing_jobs property to qiita_db.software.Command to easily retrieve all jobs in the system that have ran the given command.
+* Fixed the following issues: [#3068](https://github.com/qiita-spots/qiita/issues/3068), [#3072](https://github.com/qiita-spots/qiita/issues/3072), [#3076](https://github.com/qiita-spots/qiita/issues/3076), and [#3070](https://github.com/qiita-spots/qiita/issues/3070).
 
 Version 2021.03
 ---------------

README.rst

@@ -3,7 +3,7 @@ Qiita (canonically pronounced *cheetah*)
 
 |Build Status| |Coverage Status|
 
-Advances in sequencing, proteomics, transcriptomics and metabolomics are giving
+Advances in sequencing, proteomics, transcriptomics, metabolomics, and others are giving
 us new insights into the microbial world and dramatically improving our ability
 to understand their community composition and function at high resolution.
 These new technologies are generating vast amounts of data, even from a single
@@ -17,16 +17,16 @@ compute resources to the global community, alleviating the technical burdens,
 such as familiarity with the command line or access to compute power, that are
 typically limiting for researchers studying microbial ecology.
 
-Qiita is currently in beta status. We are very open to community
+Qiita is currently in production/stable status. We are very open to community
 contributions and feedback. If you're interested in contributing to Qiita,
 see `CONTRIBUTING.md <https://github.com/qiita-spots/qiita/blob/master/CONTRIBUTING.md>`__.
 If you'd like to report bugs or request features, you can do that in the
 `Qiita issue tracker <https://github.com/qiita-spots/qiita/issues>`__.
 
 To install and configure your own Qiita server, see
-`INSTALL.md <https://github.com/qiita-spots/qiita/blob/master/INSTALL.md>`__.
+`INSTALL.md <https://github.com/qiita-spots/qiita/blob/master/INSTALL.md>`__. However, Qiita is not designed to be used locally but rather on a server, we therefore advise against installing your own version on a personal computer. Nevertheless, it can run just fine on a laptop or small computer for development and educational purposes. For example, for every single PR and release, we install Qiita from scratch as GitHub Actions, you can follow `these steps <https://github.com/qiita-spots/qiita/actions>`__.
 
-For more specific details about qiita visit `the Qiita main site tutorial <https://qiita.microbio.me/static/doc/html/qiita-philosophy/index.html>`__.
+For more specific details about Qiita's philosophy and design visit `the Qiita main site tutorial <https://qiita.microbio.me/static/doc/html/qiita-philosophy/index.html>`__.
 
 Current features
 ----------------
@@ -39,9 +39,9 @@ Current features
 * Easy long-term sequence data deposition to the European Nucleotide Archive (ENA),
   part of the European Bioinformatics Institute (EBI) for private and public
   studies.
-* Raw data processing for `Target Gene, Metagenomic, Metabolomic and BIOM files <https://qiita.ucsd.edu/static/doc/html/processingdata/index.html#processing-recommendations>`. BIOM files can be added as new preparation files for downstream analyses; however, this cannot be made public.
-
-* Basic downstream analyses using Qiime2.
+* Raw data processing for `Target Gene, Metagenomic, Metabolomic, Genome Isolates and BIOM files <https://qiita.ucsd.edu/static/doc/html/processingdata/index.html#processing-recommendations>`__. NOTE: BIOM files can be added as new preparation files for downstream analyses; however, this cannot be made public in the system.
+* Basic downstream analyses using QIIME 2. Note that Qiita produces qza/qzv in the analytical steps but you can also convert `non QIIME 2 artifacts <https://qiita.ucsd.edu/static/doc/html/faq.html#how-to-convert-qiita-files-to-qiime2-artifacts>`__.
+* Bulk download of `studies and artifacts <https://qiita.ucsd.edu/static/doc/html/downloading.html>`__.
 * Basic study search in the study listing page.
 * Complex metadata search via redbiom.
 
@@ -55,20 +55,7 @@ Accepted raw files
 * Multiplexed FASTQ: forward, reverse (optional), and barcodes
 * Per sample FASTQ: forward and reverse (optional)
 * Multiplexed FASTA/qual files
-
-Roadmap
--------
-
-The following is a non-exhaustive list of features that we plan to add in the
-future.
-
-* Integration of other pipelines via artifacts. Processing of raw data in
-  external sources. For example, metabolomics processing in
-  `GNPS <http://gnps.ucsd.edu>`__ and data visualization in Qiita.
-* Creation of a REST API to query and access the data hosted by Qiita.
-* Improved analysis pipeline for target gene datasets.
-* Crowd-sourced metadata curation of existing studies: improve the metadata of
-  existing studies by submitting a fix proposals to the authors of the study.
+* Per sample FASTA, only for "Full Length Operon"
 
 
 .. |Build Status| image:: https://github.com/qiita-spots/qiita/actions/workflows/qiita-ci.yml/badge.svg

qiita_core/__init__.py

@@ -6,4 +6,4 @@
 # The full license is in the file LICENSE, distributed with this software.
 # -----------------------------------------------------------------------------
 
-__version__ = "2021.03"
+__version__ = "2021.05"

qiita_db/__init__.py

@@ -27,7 +27,7 @@
 from . import user
 from . import processing_job
 
-__version__ = "2021.03"
+__version__ = "2021.05"
 
 __all__ = ["analysis", "artifact",  "archive", "base", "commands",
            "environment_manager", "exceptions", "investigation", "logger",

qiita_db/artifact.py

@@ -1214,8 +1214,10 @@ def _add_edge(edges, src, dest):
                                 nodes[job.id] = ('job', job)
 
                     elif n_type == 'job':
-                        # Ignore the generate summary jobs
-                        if n_obj.command.name == 'Generate HTML summary':
+                        # Ignore the generate summary and delete jobs
+                        # (they don't create new artifacts)
+                        if n_obj.command.name in {'Generate HTML summary',
+                                                  'delete_artifact'}:
                             continue
                         jstatus = n_obj.status
                         # If the job is in success we don't need to do anything
@@ -1475,3 +1477,44 @@ def jobs(self, cmd=None, status=None, show_hidden=False):
             qdb.sql_connection.TRN.add(sql, sql_args)
             return [qdb.processing_job.ProcessingJob(jid)
                     for jid in qdb.sql_connection.TRN.execute_fetchflatten()]
+
+    @property
+    def get_commands(self):
+        """Returns the active commands that can process this kind of artifact
+
+        Returns
+        -------
+        list of qiita_db.software.Command
+            The commands that can process the given artifact tyoes
+        """
+        dws = []
+        with qdb.sql_connection.TRN:
+            # get all the possible commands
+            sql = """SELECT DISTINCT qiita.command_parameter.command_id
+                     FROM qiita.artifact
+                     JOIN qiita.parameter_artifact_type
+                        USING (artifact_type_id)
+                     JOIN qiita.command_parameter USING (command_parameter_id)
+                     JOIN qiita.software_command ON (
+                        qiita.command_parameter.command_id =
+                        qiita.software_command.command_id)
+                     WHERE artifact_id = %s AND active = True"""
+            if self.analysis is None:
+                sql += " AND is_analysis = False"
+                # get the workflows that match this artifact so we can filter
+                # the available commands based on the commands in the worflows
+                # for that artifact
+                dws = [w for w in qdb.software.DefaultWorkflow.iter()
+                       if self.data_type in w.data_type]
+            else:
+                sql += " AND is_analysis = True"
+
+            qdb.sql_connection.TRN.add(sql, [self.id])
+            cids = set(qdb.sql_connection.TRN.execute_fetchflatten())
+
+            if dws:
+                cmds = {n.default_parameter.command.id
+                        for w in dws for n in w.graph.nodes}
+                cids = cmds & cids
+
+            return [qdb.software.Command(cid) for cid in cids]

qiita_db/processing_job.py

@@ -1284,7 +1284,7 @@ def _complete_artifact_transformation(self, artifacts_data):
                 if naming_params:
                     params = self.parameters.values
                     art_name = "%s %s" % (
-                        out_name, ' '.join([str(params[p])
+                        out_name, ' '.join([str(params[p]).split('/')[-1]
                                             for p in naming_params]))
                 else:
                     art_name = out_name

qiita_db/software.py

@@ -63,7 +63,7 @@ class Command(qdb.base.QiitaObject):
 
     @classmethod
     def get_commands_by_input_type(cls, artifact_types, active_only=True,
-                                   exclude_analysis=True):
+                                   exclude_analysis=True, prep_type=None):
         """Returns the commands that can process the given artifact types
 
         Parameters
@@ -94,8 +94,17 @@ def get_commands_by_input_type(cls, artifact_types, active_only=True,
             if exclude_analysis:
                 sql += " AND is_analysis = False"
             qdb.sql_connection.TRN.add(sql, [tuple(artifact_types)])
-            for c_id in qdb.sql_connection.TRN.execute_fetchflatten():
-                yield cls(c_id)
+            cids = set(qdb.sql_connection.TRN.execute_fetchflatten())
+
+            if prep_type is not None:
+                dws = [w for w in qdb.software.DefaultWorkflow.iter()
+                       if prep_type in w.data_type]
+                if dws:
+                    cmds = {n.default_parameter.command.id
+                            for w in dws for n in w.graph.nodes}
+                    cids = cmds & cids
+
+            return [cls(cid) for cid in cids]
 
     @classmethod
     def get_html_generator(cls, artifact_type):
@@ -425,10 +434,14 @@ def create(cls, software, name, description, parameters, outputs=None,
                              qdb.util.convert_to_id(at[0], 'artifact_type'),
                              at[1]])
                     else:
-                        sql_args.append(
-                            [pname, c_id,
-                             qdb.util.convert_to_id(at, 'artifact_type'),
-                             False])
+                        try:
+                            at_id = qdb.util.convert_to_id(at, 'artifact_type')
+                        except qdb.exceptions.QiitaDBLookupError:
+                            msg = (f'Error creating {software.name}, {name}, '
+                                   f'{description} - Unknown artifact_type: '
+                                   f'{at}')
+                            raise ValueError(msg)
+                        sql_args.append([pname, c_id, at_id, False])
 
                 sql = """INSERT INTO qiita.command_output
                             (name, command_id, artifact_type_id,

qiita_db/support_files/qiita-db.html

@@ -603,6 +603,22 @@ def test_jobs_cmd_and_status(self):
         exp = []
         self.assertEqual(obs, exp)
 
+    def test_get_commands(self):
+        # we will check only ids for simplicity
+        # checking processing artifacts
+        obs = [c.id for c in qdb.artifact.Artifact(1).get_commands]
+        self.assertEqual(obs, [1])
+        obs = [c.id for c in qdb.artifact.Artifact(2).get_commands]
+        self.assertEqual(obs, [3])
+        # this is a biom in processing, so no commands should be available
+        obs = [c.id for c in qdb.artifact.Artifact(6).get_commands]
+        self.assertEqual(obs, [])
+
+        # checking analysis object - this is a biom in analysis, several
+        # commands should be available
+        obs = [c.id for c in qdb.artifact.Artifact(8).get_commands]
+        self.assertEqual(obs, [9, 10, 11, 12])
+
 
 @qiita_test_checker()
 class ArtifactTests(TestCase):

qiita_db/test/test_artifact.py

@@ -74,6 +74,18 @@ def test_get_commands_by_input_type(self):
         exp = [qdb.software.Command(1), qdb.software.Command(2), new_cmd]
         self.assertCountEqual(obs, exp)
 
+        obs = list(qdb.software.Command.get_commands_by_input_type(
+            ['FASTQ'], active_only=False, exclude_analysis=False,
+            prep_type='Metagenomic'))
+        exp = [qdb.software.Command(1), new_cmd]
+        self.assertCountEqual(obs, exp)
+
+        obs = list(qdb.software.Command.get_commands_by_input_type(
+            ['FASTQ'], active_only=False, exclude_analysis=False,
+            prep_type='18S'))
+        exp = [qdb.software.Command(1)]
+        self.assertCountEqual(obs, exp)
+
     def test_get_html_artifact(self):
         with self.assertRaises(qdb.exceptions.QiitaDBError):
             qdb.software.Command.get_html_generator('BIOM')
@@ -329,6 +341,16 @@ def test_create_error(self):
                 "This is a command for testing", self.parameters,
                 self.outputs)
 
+        # the output type doesn't exist
+        with self.assertRaisesRegex(ValueError, "Error creating QIIME, Split "
+                                    "libraries - wrong output, This is a "
+                                    "command for testing - Unknown "
+                                    "artifact_type: BLA!"):
+            qdb.software.Command.create(
+                self.software, "Split libraries - wrong output",
+                "This is a command for testing", self.parameters,
+                {'out': 'BLA!'})
+
     def test_create(self):
         # let's deactivate all current plugins and commands; this is not
         # important to test the creation but it is important to test if a