==================
Uploading packages
==================

.. _workflow-package-upload:

Workflow ``package_upload``
===========================

This workflow signs and uploads source and/or binary packages to an upload
queue.  It is normally expected to be used as a sub-workflow.

* ``task_data``:

  * ``source_artifact`` (:ref:`lookup-single`, optional): a
    ``debian:source-package`` or ``debian:upload`` artifact representing the
    source package (the former is used when the workflow is started based on
    a ``.dsc`` rather than a ``.changes``)
  * ``binary_artifacts`` (:ref:`lookup-multiple`, optional): a list of
    ``debian:upload`` artifacts representing the binary packages
  * ``merge_uploads`` (boolean, defaults to False): if True, merge the
    uploads and create a single ``PackageUpload`` task to upload them all
    together; if False, create a separate ``PackageUpload`` task for each
    upload
  * ``since_version`` (string, optional): passed to
    :ref:`task-make-source-package-upload` if ``source_artifact`` is a
    ``debian:source-package``
  * ``target_distribution`` (string, optional): passed to
    :ref:`task-make-source-package-upload` if ``source_artifact`` is a
    ``debian:source-package``
  * ``key`` (:ref:`lookup-single`, optional): the ``debusine:signing-key``
    artifact to sign the upload with, which must have purpose ``openpgp``
  * ``require_signature`` (boolean, defaults to True): whether the upload
    must be signed
  * ``target`` (required): the upload queue, as an ``ftp://`` or ``sftp://``
    URL
  * ``vendor`` (string, optional): the distribution vendor to use for running
    :ref:`task-make-source-package-upload` and :ref:`task-merge-uploads`
  * ``codename`` (string, optional): the distribution codename to use for
    running :ref:`task-make-source-package-upload` and :ref:`task-merge-uploads`

At least one of ``source_artifact`` and ``binary_artifacts`` must be set.

The workflow creates the following tasks, each of which has a dependency on
the previous one in sequence, using event reactions to store output in the
workflow's internal collection for use by later tasks:

* if ``source_artifact`` is a ``debian:source-package`` artifact: a
  :ref:`task-make-source-package-upload` (with ``since_version`` and
  ``target_distribution``) to build a corresponding ``.changes`` file
  Uses ``vendor`` and ``codename`` to construct the environment lookup.
* if ``merge_uploads`` is True and there is more than one source and/or
  binary artifact: a :ref:`task-merge-uploads` to combine them into a single
  upload. Uses ``vendor`` and ``codename`` to construct the environment lookup.
* for each upload (or for the single merged upload, if merging):

  * if ``key`` is provided: a :ref:`task-debsign` to have debusine sign the
    upload with the given key
  * if ``key`` is not provided and ``require_signature`` is True: an
    :ref:`task-external-debsign` to wait until a user provides a signature,
    which debusine will then include with the upload
  * a :ref:`task-package-upload`, to upload the result to the given upload
    queue

UI changes
==========

Wait tasks will generally require some form of UI change.  In particular,
when showing a blocked ``ExternalDebsign`` work request, the web UI advises
the user to run a new client command (e.g. ``debusine provide-signature
<work request ID>``, which downloads the unsigned upload, signs it, and
uploads it back to the ``external-debsign`` API view above.

Pipeline considerations
=======================

The :ref:`package_upload workflow <workflow-package-upload>` will typically
be used as a sub-workflow of a smart "pipeline" workflow.  There are three
main use cases:

* Upload a source package to debusine, have it tested, and then have
  debusine pass on that upload to an external upload queue.

  In this case, the ``package_upload`` workflow's task data should set
  ``source_artifact`` to the source package and leave ``binary_artifacts``
  empty.

* Upload a source package to debusine, have it tested, and then have
  debusine upload both the source and all built binaries to an external
  upload queue.  (For example, this is useful when uploading a package to
  Debian that will land in the NEW queue, since Debian currently requires
  binaries for NEW uploads.)

  In this case, the ``package_upload`` workflow's task data should set
  ``source_artifact`` to the source package, set ``binary_artifacts`` to a
  list of :ref:`single lookups <lookup-single>` matching each of the binary
  uploads from the super-workflow's internal collection (e.g.
  ``[internal@collections/name:build-all,
  internal@collections/name:build-amd64]``, and set ``merge_uploads`` to
  True.

* debusine acts as a build daemon, building a source package for a number of
  architectures and uploading each of them as soon as the builds finish.

  In this case, the ``package_upload`` workflow's task data should leave
  ``source_artifact`` unset, set ``binary_artifacts`` to a list of
  :ref:`single lookups <lookup-single>` matching each of the binary uploads
  from the super-workflow's internal collection, and set ``merge_uploads``
  to False.

Generic code for creating child work requests using artifacts from the
workflow's internal collection adds appropriate dependencies on the work
requests that are expected to provide those artifacts.

If the parent workflow needs some kind of manual validation step to complete
before starting the upload (typically in the case of manual uploads but not
when acting as a build daemon), it should add a dependency from the
``package_upload`` sub-workflow to the validation workflow step.  The
``package_upload`` sub-workflow will be populated before validation is
complete (since the root workflow handles population of all its
sub-workflows), but it will not start running until validation is complete.
