Streams API

Skeleton

Examples

<stream
    id="some-family-stream"
    bucket="family(648ab40ba34e015fa57ee579, {{page.section.id}}, $$any, $$none, $$empty, None)"
    use_pins="true"
    reverse_order="false"
/>
<posts
    source="stream"
    stream_id="some-family-stream"
    ...
/>
<stream
    id="some-family-stream"
    bucket="family(648ab40ba34e015fa57ee579, {{page.section.id}}, $$any, $$none, $$empty, None)"
    use_pins="false"
    reverse_order="true"
>
    <period type="window" size="7" unit_in_seconds="{{ 86400 }}" />
/>
<posts
    source="stream"
    stream_id="some-family-stream"
    ...
/>
<stream
    id="some-family-stream"
    bucket="family(648ab40ba34e015fa57ee579, {{page.section.id}}, $$any, $$none, $$empty, None)"
>
    <period type="lifetime" />
/>
<posts
    source="stream"
    stream_id="some-family-stream"
    ...
/>
<posts
    source="stream"
    bucket="family(648ab40ba34e015fa57ee579, {{page.section.id}}, $$any, $$none, $$empty, None)"
    use_pins="false"
    ...
/>

Attributes

  • stream.id - the stream configuration ID which is unique to the layouts

  • stream.bucket - the stream bucket reference

    • family(family_id, arg1, arg2, arg3…) – references a particular bucket in a family

  • stream.use_pins - whether to use pins or not (default: false)

  • stream.period.type – the type of period to use for the stream

    • window – a window of time

      • stream.period.size – the size of the window in units

      • stream.period.unit_in_seconds – the unit of time in seconds

    • lifetime – the lifetime

  • posts.stream_id – the stream configuration ID to use for the posts

  • posts.bucket – the stream bucket reference to use for the posts (@see stream.bucket)

  • posts.use_pins - whether to use pins or not (@see stream.use_pins)

Endpoints

Get Families

GET /core/v1/streams/families

200 OK

{
  "families": [
    {
      "id": "648ab3c9522b443bd06774d1",
      "title": null,  // or "Some interesting posts [v1.0]",
      "entity": <entity>,
      "status": <family status>,
      "filter": null,  // or <family filter configuration>
      "scores": null  // or <family scores configuration>
    }
  ]
}

Hints:

Create Family

POST /core/v1/streams/families
{
  "entity": <entity>,
  "commands": [
    <family command description>,
    <family command description>,
    ...
  ]
}

200 OK

{
  "family": {
    "id": "648ab3c9522b443bd06774d1",
    "title": null,
    "entity": <entity>,
    "status": <family status>,
    "filter": null,  // or <family filter configuration>
    "scores": null  // or <family scores configuration>
  }
}

Hints:

Update Family

PUT /core/v1/streams/families/648ab3c9522b443bd06774d1
{
  "commands": [
    <family command description>,
    <family command description>,
    ...
  ]
}

200 OK

{
  "family": {
    "id": "648ab3c9522b443bd06774d1",
    "title": null,  // or "Some interesting posts [v1.0]",
    "entity": <entity>,
    "status": <family status>,
    "filter": null,  // or <family filter configuration>
    "scores": null  // or <family scores configuration>
  }
}

Hints:

Reindex Family

PUT /core/v1/streams/families/648ab3c9522b443bd06774d1/reindex?should_reset_cache=true
GET /core/v1/streams/families/648ab3c9522b443bd06774d1/reindex?task_id=38e9eea7-fc5b-4373-90f0-cb2d59109113

200 OK

{
  "task": {
    "id": "38e9eea7-fc5b-4373-90f0-cb2d59109113",
    "is_ready": true,
    "progress": null,  // or same as "response"
    "response": {
      "entities": {
        "total_count": 12343,
        "indexed_count": 12343
      }
    },
    "exception": null
}

Delete Family

DELETE /core/v1/streams/families/648ab3c9522b443bd06774d1

200 OK

{}

Find Stream Records

GET /core/v1/streams/records?bucket=<bucket string>&limit=<limit>[&offset=<offset>[&cursor=<string>[&mode=default|ignore-pins|ignore-pins-and-reverse-order]]]

200 OK

{
  "records": [
    {
      "entity": <entity>,
      "cursor": <string>  // or `null` if the entity is pinned in this position
    },
    ...
  ]
}

Hints:

  • <entity> (see: Entities)

  • <bucket string> (e.g., family(648ab40ba34e015fa57ee579, <some_section_id>, $$any, $$none, $$empty, None))

Count Stream Records

GET /core/v1/streams/records/count?bucket=<bucket string>

200 OK

{
  "count": 592
}

Hints:

  • <bucket string> (e.g., family(648ab40ba34e015fa57ee579, <some_section_id>, $$any, $$none, $$empty, None))

Update Stream Records

POST /core/v1/streams/records?bucket=<bucket string>

200 OK

{
  "commands": [
    <stream command>,
    <stream command>,
    ...
  ]
}

Hints:

  • <bucket string> (e.g., family(648ab40ba34e015fa57ee579, <some_section_id>, $$any, $$none, $$empty, None))

  • <stream command> (see: Stream Commands)

Families

Family of Streams

Statuses

  • {"paused": {}} – the family is paused. Allows to configure the family and remove it

  • {"active": {}} – the family is active. Allows entities to be routed by the family configuration into appropriate buckets in the required order

Commands

  1. Configure family

    • Set Title

      {"set-title": "Some less interesting posts [v1.1]"}

    • Set Filter

      {"set-filter": <family filter configuration>}

    • Set Scores

      {"set-scores": <family scores configuration>}

  2. Manage family status

    • Activate

      {"activate": {}}

    • Pause

      {"pause": {}}}

Hints:

Filters

Taxonomy

{"taxonomy": {
    variables: [
        <variable configuration>,
        <variable configuration>,
        ...
    ],
    filters: [
        <filter>,
        <filter>,
        <filter>,
        ...
    ]
}}
Variables
{
  "alias": null,  // or "some_alias", not implemented yet
  "scope": <scope>,
  "yields": [
    "<yield>",
    "<yield>",
    ...
  ]
}

Yields:

  • "$any" – route the entity to buckets where this variable is ignored (e.g. post.sections not empty –> an $any bucket)

  • "$none" – route the entity to buckets where this variable is either not set or represent an empty lists (e.g. post.sections is empty –> a $none bucket)

  • "$each" – route the entity to buckets where this variable takes one of the values from the variable scope (e.g. post.sections –> N section_id buckets)

Scopes:

  • {"post-authors": {}} – yields indexed post author IDs

  • {"post-primary-tags": {}} – yields indexed post primary tag slug

  • {"post-regular-tags": {}} – yields indexed post regular tag slugs

  • {"post-primary-sections": {}} – yields indexed post primary section ID

  • {"post-regular-sections": {}} – yields indexed post regular section IDs

  • {"post-custom-field-values": {"path": "path.to.field"}} – yields indexed post custom field values

  • {"site-sections": {"filters": [<section filter>, <section filter>, ...]}} – yields indexed post primary topic ID

  • {"manual": {"state": <variable state>}} – yields manually added values

States:

  • {"tags": {"slugs": [<tag_slug>, <tag_slug>, ...]}} – yields manually added tag slugs

  • {"sections": {"ids": [<section_id>, <section_id>, ...]}} – yields manually added section IDs

  • {"scalars": {"type": "str" | "int" | "float", "values": [<value>, <value>, ...]}} – yields manually added scalar values

Filters

Variable filters allow to distribute entities into buckets based on the values of the variables. The following variable filters are supported:

  • Resolve variable reference to a set of taxonomy filters

{"$switch": {
  "$ref": "$$1", // or "$some_alias" (not implemented yet),
  "$cases": {
    "$any": [ // an entity goes to this bucket no matter the variable value
      <variable taxonomy filter>,
      ...
    ],
    "$none": [ // an entity goes to this bucket if the variable value is not set or is an empty list
      <variable taxonomy filter>,
      ...
    ],
    "$each": [ // an entity goes to a bucket for each value the variable yields
      {"any_label": [
        {"$return": {"value": {
          "field": <taxonomy field>,
          "value": "$$1"  // or "$some_alias" (not implemented yet)
        }}},
        <variable taxonomy label>,
        ...
      ]},
      {"any_range": [
        {"$return": {
          "field": <taxonomy field>,
          "min": "$$1[1]",  // or "$some_alias[1]" (not implemented yet)
          "max": "$$1[2]"  // or "$some_alias[2]" (not implemented yet)
        }},
        <variable taxonomy range>,
        ...
      ]},
      <variable taxonomy filter>,
      ...
    ]
  }
}}
  • Return a set of taxonomy filters without applying any variables

{"$return": [
  <variable taxonomy filter>,
  <variable taxonomy filter>,
  ...
]}

Hints:

  • <variable taxonomy filter> (e.g., {“any_label”: {“$return”: [<taxonomy label>, …]}})

  • <variable taxonomy label> (e.g., {“$return”: <taxonomy label>})

  • <variable taxonomy range> (e.g., {“$return”: <taxonomy range>})

  • <taxonomy filter> (see: Filters)

  • <taxonomy label> (see: Labels)

  • <taxonomy range> (see: Ranges)

  • <taxonomy field> (see: Fields)

Scores

Static

{
    "static": {
        "mode": <mode>,
        "field": <taxonomy field>,
        "scale": null  // or a float number to scale the score taken from the entity field
    }}
}

Modes:

  • {"init": {}} – when an entity indexed for the first time calculate the value and never update it

  • {"track": {}} – when an entity indexed for the first time calculate the value and update it when the entity is updated

  • {"track-until-moved": {}} – when an entity indexed for the first time calculate the value and update it when the entity is updated. Stop tracking the score for a particular bucket as soon as the entity is manually moved in that bucket

Incremental

{
    "incremental": {
        "period": <period>,
        "weights": [
            {"event": <taxonomy-event>, "weight": <weight>}
        ]
    }}
}

Periods:

  • {"window": {"size": 7, "unit_in_seconds": 86400}} – a window of time, e.g. 7 days

  • {"lifetime": {}} – the lifetime

Hints:

  • <taxonomy field> (see: Fields)

  • <taxonomy event> (see: Events)

Streams

Commands

  • Pin entity

    {"pin": {"entity": <entity>, "position": <position>}}

  • Unpin entity

    {"unpin": {"entity": <entity>}}

  • Move entity to position

    {"move-to": {"entity": <entity>, "position": <position>}}

  • Move entity after another entity

    {"move-after": {"entity": <entity>, "other_entity": <entity>}}

Hints:

  • <entity> (see: Entities)

  • <position> – a numeric rank in the stream starting with 0