:orphan:

.. role:: json(code)
   :language: json

.. role:: python(code)
   :language: python


Streams API
===========

.. contents:: :depth: 3


Skeleton
########

Examples
--------

.. code:: xml

    <stream
        id="some-family-stream"
        bucket="family(648ab40ba34e015fa57ee579, {{page.section.id}}, $$any, $$none, $$empty, None)"
        use_pins="true"
        reverse_order="false"
    />
    <posts
        source="stream"
        stream_id="some-family-stream"
        ...
    />

.. code:: xml

    <stream
        id="some-family-stream"
        bucket="family(648ab40ba34e015fa57ee579, {{page.section.id}}, $$any, $$none, $$empty, None)"
        use_pins="false"
        reverse_order="true"
    >
        <period type="window" size="7" unit_in_seconds="{{ 86400 }}" />
    />
    <posts
        source="stream"
        stream_id="some-family-stream"
        ...
    />

.. code:: xml

    <stream
        id="some-family-stream"
        bucket="family(648ab40ba34e015fa57ee579, {{page.section.id}}, $$any, $$none, $$empty, None)"
    >
        <period type="lifetime" />
    />
    <posts
        source="stream"
        stream_id="some-family-stream"
        ...
    />

.. code:: xml

    <posts
        source="stream"
        bucket="family(648ab40ba34e015fa57ee579, {{page.section.id}}, $$any, $$none, $$empty, None)"
        use_pins="false"
        ...
    />


Attributes
----------

- ``stream.id`` - the stream configuration ID which is unique to the layouts
- ``stream.bucket`` - the stream bucket reference

  - *family(family_id, arg1, arg2, arg3...)* -- references a particular bucket in a family

- ``stream.use_pins`` - whether to use pins or not (default: ``false``)
- ``stream.period.type`` -- the type of period to use for the stream

  - *window* -- a window of time

    - ``stream.period.size`` -- the size of the window in units
    - ``stream.period.unit_in_seconds`` -- the unit of time in seconds

  - *lifetime* -- the lifetime

- ``posts.stream_id`` -- the stream configuration ID to use for the posts
- ``posts.bucket`` -- the stream bucket reference to use for the posts (@see ``stream.bucket``)
- ``posts.use_pins`` - whether to use pins or not (@see ``stream.use_pins``)


Endpoints
#########

Get Families
------------

.. code::

    GET /core/v1/streams/families

200 OK
******

.. code:: json

    {
      "families": [
        {
          "id": "648ab3c9522b443bd06774d1",
          "title": null,  // or "Some interesting posts [v1.0]",
          "entity": <entity>,
          "status": <family status>,
          "filter": null,  // or <family filter configuration>
          "scores": null  // or <family scores configuration>
        }
      ]
    }

Hints:

- <entity> (see: :ref:`taxonomy-entities`)
- <family status> (see: `Family Statuses`_)
- <family filter configuration> (see: `Family Filter Configuration`_)
- <family scores configuration> (see: `Family Scores Configuration`_)


Create Family
-------------

.. code::

    POST /core/v1/streams/families

.. code:: json

    {
      "entity": <entity>,
      "commands": [
        <family command description>,
        <family command description>,
        ...
      ]
    }

200 OK
******

.. code:: json

    {
      "family": {
        "id": "648ab3c9522b443bd06774d1",
        "title": null,
        "entity": <entity>,
        "status": <family status>,
        "filter": null,  // or <family filter configuration>
        "scores": null  // or <family scores configuration>
      }
    }

Hints:

* <entity> (see: :ref:`taxonomy-entities`)
* <family command description> (see: `Family Commands`_)
* <family status> (see: `Family Statuses`_)
* <family filter configuration> (see: `Family Filter Configuration`_)
* <family scores configuration> (see: `Family Scores Configuration`_)


Update Family
-------------

.. code::

    PUT /core/v1/streams/families/648ab3c9522b443bd06774d1

.. code:: json

    {
      "commands": [
        <family command description>,
        <family command description>,
        ...
      ]
    }

200 OK
******

.. code:: json

    {
      "family": {
        "id": "648ab3c9522b443bd06774d1",
        "title": null,  // or "Some interesting posts [v1.0]",
        "entity": <entity>,
        "status": <family status>,
        "filter": null,  // or <family filter configuration>
        "scores": null  // or <family scores configuration>
      }
    }

Hints:

* <entity> (see: :ref:`taxonomy-entities`)
* <family command description> (see: `Family Commands`_)
* <family status> (see: `Family Statuses`_)
* <family filter configuration> (see: `Family Filter Configuration`_)
* <family scores configuration> (see: `Family Scores Configuration`_)


Reindex Family
--------------

.. code::

    PUT /core/v1/streams/families/648ab3c9522b443bd06774d1/reindex?should_reset_cache=true

.. code::

    GET /core/v1/streams/families/648ab3c9522b443bd06774d1/reindex?task_id=38e9eea7-fc5b-4373-90f0-cb2d59109113

200 OK
******

.. code:: json

    {
      "task": {
        "id": "38e9eea7-fc5b-4373-90f0-cb2d59109113",
        "is_ready": true,
        "progress": null,  // or same as "response"
        "response": {
          "entities": {
            "total_count": 12343,
            "indexed_count": 12343
          }
        },
        "exception": null
    }


Delete Family
-------------

.. code::

    DELETE /core/v1/streams/families/648ab3c9522b443bd06774d1

200 OK
******

.. code:: json

    {}


Find Stream Records
-------------------

.. code::

    GET /core/v1/streams/records?bucket=<bucket string>&limit=<limit>[&offset=<offset>[&cursor=<string>[&mode=default|ignore-pins|ignore-pins-and-reverse-order]]]

200 OK
******

.. code:: json

    {
      "records": [
        {
          "entity": <entity>,
          "cursor": <string>  // or `null` if the entity is pinned in this position
        },
        ...
      ]
    }

Hints:

- <entity> (see: :ref:`taxonomy-entities`)
- <bucket string> (e.g., `family(648ab40ba34e015fa57ee579, <some_section_id>, $$any, $$none, $$empty, None)`)


Count Stream Records
--------------------

.. code::

    GET /core/v1/streams/records/count?bucket=<bucket string>

200 OK
******

.. code:: json

    {
      "count": 592
    }

Hints:

- <bucket string> (e.g., `family(648ab40ba34e015fa57ee579, <some_section_id>, $$any, $$none, $$empty, None)`)


Update Stream Records
---------------------

.. code::

    POST /core/v1/streams/records?bucket=<bucket string>

200 OK
******

.. code:: json

    {
      "commands": [
        <stream command>,
        <stream command>,
        ...
      ]
    }

Hints:

- <bucket string> (e.g., `family(648ab40ba34e015fa57ee579, <some_section_id>, $$any, $$none, $$empty, None)`)
- <stream command> (see: `Stream Commands`_)


Families
########

.. image:: /_static/images/content/streams/family-concept.png
  :alt: Family of Streams


.. _Family Statuses:

Statuses
--------

* ``{"paused": {}}`` -- the family is paused. Allows to configure the family and remove it
* ``{"active": {}}`` -- the family is active. Allows entities to be routed by the family configuration into appropriate buckets in the required order


.. _Family Commands:

Commands
--------

#. Configure family

    * Set Title
        :json:`{"set-title": "Some less interesting posts [v1.1]"}`
    * Set Filter
        :json:`{"set-filter": <family filter configuration>}`
    * Set Scores
        :json:`{"set-scores": <family scores configuration>}`

#. Manage family status

    * Activate
        :json:`{"activate": {}}`
    * Pause
        :json:`{"pause": {}}}`

Hints:

* <family filter configuration> (see: `Family Filter Configuration`_)
* <family scores configuration> (see: `Family Scores Configuration`_)

.. _Family Filter Configuration:

Filters
-------

Taxonomy
********

.. code:: json

    {"taxonomy": {
        variables: [
            <variable configuration>,
            <variable configuration>,
            ...
        ],
        filters: [
            <filter>,
            <filter>,
            <filter>,
            ...
        ]
    }}

Variables
~~~~~~~~~

.. code:: json

    {
      "alias": null,  // or "some_alias", not implemented yet
      "scope": <scope>,
      "yields": [
        "<yield>",
        "<yield>",
        ...
      ]
    }

Yields:

* :json:`"$any"` -- route the entity to buckets where this variable is ignored (e.g. `post.sections` not empty --> an `$any` bucket)
* :json:`"$none"` -- route the entity to buckets where this variable is either not set or represent an empty lists (e.g. `post.sections` is empty --> a `$none` bucket)
* :json:`"$each"` -- route the entity to buckets where this variable takes one of the values from the variable scope (e.g. `post.sections` --> N `section_id` buckets)

Scopes:

* :json:`{"post-authors": {}}` -- yields indexed post author IDs
* :json:`{"post-communities": {}}` -- yields indexed post community IDs
* :json:`{"post-primary-tags": {}}` -- yields indexed post primary tag slug
* :json:`{"post-regular-tags": {}}` -- yields indexed post regular tag slugs
* :json:`{"post-primary-sections": {}}` -- yields indexed post primary section ID
* :json:`{"post-regular-sections": {}}` -- yields indexed post regular section IDs
* :json:`{"post-custom-field-values": {"path": "path.to.field"}}` -- yields indexed post custom field values
* :json:`{"site-sections": {"filters": [<section filter>, <section filter>, ...]}}` -- yields indexed post primary topic ID
* :json:`{"manual": {"state": <variable state>}}` -- yields manually added values

States:

* :json:`{"tags": {"slugs": [<tag_slug>, <tag_slug>, ...]}}` -- yields manually added tag slugs
* :json:`{"sections": {"ids": [<section_id>, <section_id>, ...]}}` -- yields manually added section IDs
* :json:`{"scalars": {"type": "str" | "int" | "float", "values": [<value>, <value>, ...]}}` -- yields manually added scalar values


Filters
~~~~~~~

Variable filters allow to distribute entities into buckets based on the values of the variables. The following variable filters are supported:

- Resolve variable reference to a set of taxonomy filters

.. code:: json

    {"$switch": {
      "$ref": "$$1", // or "$some_alias" (not implemented yet),
      "$cases": {
        "$any": [ // an entity goes to this bucket no matter the variable value
          <variable taxonomy filter>,
          ...
        ],
        "$none": [ // an entity goes to this bucket if the variable value is not set or is an empty list
          <variable taxonomy filter>,
          ...
        ],
        "$each": [ // an entity goes to a bucket for each value the variable yields
          {"any_label": [
            {"$return": {"value": {
              "field": <taxonomy field>,
              "value": "$$1"  // or "$some_alias" (not implemented yet)
            }}},
            <variable taxonomy label>,
            ...
          ]},
          {"any_range": [
            {"$return": {
              "field": <taxonomy field>,
              "min": "$$1[1]",  // or "$some_alias[1]" (not implemented yet)
              "max": "$$1[2]"  // or "$some_alias[2]" (not implemented yet)
            }},
            <variable taxonomy range>,
            ...
          ]},
          <variable taxonomy filter>,
          ...
        ]
      }
    }}

- Return a set of taxonomy filters without applying any variables

.. code:: json

    {"$return": [
      <variable taxonomy filter>,
      <variable taxonomy filter>,
      ...
    ]}

Hints:

- <variable taxonomy filter> (e.g., `{"any_label": {"$return": [<taxonomy label>, ...]}}`)
- <variable taxonomy label> (e.g., `{"$return": <taxonomy label>}`)
- <variable taxonomy range> (e.g., `{"$return": <taxonomy range>}`)
- <taxonomy filter> (see: :ref:`taxonomy-filters`)
- <taxonomy label> (see: :ref:`taxonomy-labels`)
- <taxonomy range> (see: :ref:`taxonomy-ranges`)
- <taxonomy field> (see: :ref:`taxonomy-entity-fields`)


.. _Family Scores Configuration:

Scores
------

Static
******

.. code:: json

    {
        "static": {
            "mode": <mode>,
            "field": <taxonomy field>,
            "scale": null  // or a float number to scale the score taken from the entity field
        }}
    }

Modes:

- :json:`{"init": {}}` -- when an entity indexed for the first time calculate the value and never update it
- :json:`{"track": {}}` -- when an entity indexed for the first time calculate the value and update it when the entity is updated
- :json:`{"track-until-moved": {}}` -- when an entity indexed for the first time calculate the value and update it when the entity is updated. Stop tracking the score for a particular bucket as soon as the entity is manually moved in that bucket

Incremental
***********

.. code:: json

    {
        "incremental": {
            "period": <period>,
            "weights": [
                {"event": <taxonomy-event>, "weight": <weight>}
            ]
        }}
    }

Periods:

- :json:`{"window": {"size": 7, "unit_in_seconds": 86400}}` -- a window of time, e.g. 7 days
- :json:`{"lifetime": {}}` -- the lifetime

Hints:

- <taxonomy field> (see: :ref:`taxonomy-entity-fields`)
- <taxonomy event> (see: :ref:`taxonomy-entity-events`)


Streams
########


.. _Stream Commands:

Commands
--------

* Pin entity
    :json:`{"pin": {"entity": <entity>, "position": <position>}}`
* Unpin entity
    :json:`{"unpin": {"entity": <entity>}}`
* Move entity to position
    :json:`{"move-to": {"entity": <entity>, "position": <position>}}`
* Move entity after another entity
    :json:`{"move-after": {"entity": <entity>, "other_entity": <entity>}}`


Hints:

- <entity> (see: :ref:`taxonomy-entities`)
- <position> -- a numeric rank in the stream starting with 0