Skip to content

Recent measurements

POST {baseUrl}/v2/recent-datasource-measurements-query

By Device

This query is only by Datasource ID. To query by device or reference site, use the historical measurements report.

Returns a list of recent measurements in time descending order.

Example
url: https://clarity-data-api.clarity.io/v2/recent-datasource-measurements-query
body:

{
    "org": "myorg1234",
    "datasourceIds": ["DS123456", "DS987654"],
    "outputFrequency": "hour"
}

Cached measurements

Recent measurements are served from a fast cache that goes back in time depending on output frequency:

  • Individual measurements (minute) cached for 24 hours
  • Hourly aggregations (hour) cached for 48 hours
  • Daily aggregations (day) cached for 10 days

Continuations

A common access pattern is to routinely poll to get any new measurements since the last time you asked. It is awkward to accomplish this using startTime on your requests. Due to race conditions it is possible that you could miss a measurement, or get the same measurement twice. To make this easier, a companion endpoint accepts continuation tokens.

In your first call — using this endpoint — you specify an additional parameter: replyWithContinuationToken: true. The response will include an x-clarity-continuation-token in the header.

On subsequent calls, use the recent datasource measurements continuation endpoint and pass that continuation token back in the body (along with org). The continuation response will pick up from the last call, returning newer measurements (if any exist) for the same datasources requested in the first call. An updated continuation token will be returned as well.

You can continue making continuation queries, with the token being updated on each call. A continuation token is good for 24 hours after it is issued.

Note: If the original call was for allDatasources in the org, then the continuations will include/exclude datasources as they are subscribed/unsubscribed.

Headers

parameter description
x-api-key string The API key string.
Accept-Encoding string Encoding options, gzip is supported to compress data.

Note: We strongly recommend including it in the headers to reduce response payload size.

Example: Accept-Encoding: gzip

Request

You must include org on all requests. Provide a list of datasources in datasourceIds, or for all the organization's datasources, pass "allDatasources": true.

body parameter description
org string
required
Find the org ID by clicking on your user icon in Dashboard.
Example org ID: "myorgVD43"
allDatasources bool true to retrieve measurements for all datasources in the specified organization for which you have access.
false or omit the parameter if you want to retrict using datasourceIds.
datasourceIds array of string A list of modern datasource ids as returned from /v2/datasources.
Only measurements from periods when the datasource was subscribed by the requested org are returned.
replyWithContinuationToken bool true if you want to receive a continuation token in the response headers, otherwise omit this parameter
(see Continuations above)

And then add these parameters

additional query parameter description
outputFrequency string The output frequency of the aggregations to return in the measurement. One of:
"minute" default - returns individual measurements, with no aggregation applied.
"hour" returns 1-Hour Mean, 24-Hour Rolling Mean and NowCast aggregations, and AQIs which are calculated on these aggregations as per their standard.
"day" returns 24-Hour Mean aggregations, and AQIs which are calculated on these aggregations as per their standard.
format string The format of the response. One of:
json-long default - A JSON object per metric in a measurement
csv-wide CSV, one row per measurement
startTime string Cut-off time for the earliest measurement desired. If not specified, the start time used will be chosen based on the requested output frequency
minute → 1 hour prior to time of request
hour → 24 hours prior to time of request
day → 48 hours prior to time of request

Measurements are returned from a cache. Earlier times are accepted in the parameter, but only data from the cache will be returned.

Example: startTime: "2023-09-01T00:00:00Z"
metricSelect string Limits the metrics returned in the measurement. See Metrics selection
qcAssessment boolean When true, adds the field qcAssessment to each metric in the response. Default false. Values are valid and invalid but may be expanded in the future.
qcFlags boolean When true, adds the field qcFlags to each metric in the response. Default false. See QC Flags

Response Headers

parameter description
x-clarity-continuation-token string The continuation token, if it was requested with replyWithContinuationToken

Response Formats

json-long

A typical JSON structure including these important objects

  • request - an echo back of the original request
  • data - a list of metrics objects
  • locations - a sidebar dictionary to lookup the location of a datasource

The time field for hour or day aggregate measurements is the start of the period for the aggregate. This matches how UIs typically label hour or day aggregated information.

This example is an excerpt of a much longer response:

    "request": {
        "format": "json-long",
        "org": "myorg1234",
        "outputFrequency": "hour",
        "allDatasources": true,
        "startTime": "2023-11-03T19:00:00+00:00"
        "qcAssessment": true,
        "qcFlags": true,
    },

    "data": [
        {
            "datasourceId": "DCQYT1459"
            "time": "2023-11-03T19:00:00.000Z",
            "status": "sensor-ready",
            "value": 20.25,
            "raw": 20.25,
            "metric": "temperatureInternal1HourMean",
            "qcAssessment": "valid",
            "qcFlags": []
        },
        {
            "datasourceId": "DCQYT1459"
            "time": "2023-11-03T19:00:00.000Z",
            "status": "calibrated-ready",
            "value": 8192.31,
            "raw": 3019.89,
            "metric": "pm2_5ConcMass1HourMean",
            "qcAssessment": "invalid",
            "qcFlags": ["QC.I.OOB.001", "QC.I.OOB.002"]
        },
        ...more...
    ],

    "locations": [
        {
            "datasourceId": "DCQYT1459",
            "lon": -122.30173,
            "lat": 37.87864
        },
        {
            "datasourceId": "DDDYN3547",
            "lon": -116.7845,
            "lat": 38.28068
        },
        ...more...
    ]

Note: Best practice is that if a device in a datasource is replaced, the new device should be configured to the same location as the old device. If that does not happen, then it is possible for location to change in the middle of a time-series. If that should occur, then the locations object above will show both locations and identify the first measurement in the time-series that has the new location:

        {
            "datasourceId": "DABCD1234",
            "lon": -120.24056,
            "lat": 37.87864
        },
        {
            "datasourceId": "DABCD1234",               <--- same datasource
            "lon": -122.00052,                         <--- new location
            "lat": 38.28068,
            "changedAt": "2023-11-03T21:00:00.000Z"    <--- first measurement at new location
        },

csv-wide

Column Headers

Automation that processes this response should not assume specifc columns or column ordering. Interpret based on the header row of the response.

The response is like a CSV file: A comma-separated string with an initial header row identifying the columns and with newlines separating the measurements. The set of columns returned depends on the outputFrequency requested.

Identifying columns for all output frequencies:

column header description
datasourceId Unique identifier of the datasource. (string)
sourceId Unique identifier of the underlying device or reference site. (string)
sourceType Either CLARITY_NODE or REFERENCE_SITE. (string)
outputFrequency The output frequency; one of minute, hour, or day. (string)

Example with many metrics selected:

datasourceId,sourceId,sourceType,periodType,startOfPeriod,endOfPeriod,locationLatitude,locationLongitude,atmPressureRaw1HourMean,no2ConcCalibrated1HourMean,no2ConcCalibrated1HourMeanUsEpaAqi,no2ConcCalibratedWaDwerAqi,no2ConcRaw1HourMean,no2ConcRaw1HourMeanUsEpaAqi,no2ConcRawWaDwerAqi,o3ConcRaw1HourMean,o3ConcRawWaDwerAqi,pm10ConcMassCalibrated1HourMean,pm10ConcMassCalibrated24HourRollingMean,pm10ConcMassCalibratedWaDwerAqi,pm10ConcMassRaw1HourMean,pm10ConcMassRaw24HourRollingMean,pm10ConcMassRawWaDwerAqi,pm10ConcNumRaw1HourMean,pm10ConcNumRaw24HourRollingMean,pm1ConcMassCalibrated1HourMean,pm1ConcMassRaw1HourMean,pm1ConcNumRaw1HourMean,pm2_5ConcMassCalibrated1HourMean,pm2_5ConcMassCalibrated24HourRollingMean,pm2_5ConcMassCalibratedNowCast,pm2_5ConcMassCalibratedNowCastAqi,pm2_5ConcMassCalibratedWaDwerAqi,pm2_5ConcMassRaw1HourMean,pm2_5ConcMassRaw24HourRollingMean,pm2_5ConcMassRawNowCast,pm2_5ConcMassRawNowCastAqi,pm2_5ConcMassRawWaDwerAqi,pm2_5ConcNumRaw1HourMean,pm2_5ConcNumRaw24HourRollingMean,relHumidAmbientRaw1HourMean,relHumidInternalRaw1HourMean,temperatureAmbientRaw1HourMean,temperatureInternalRaw1HourMean,windDirectionRaw1HourMean,windSpeedRaw1HourMean
DCQYT1459,AXGCR7WJ,CLARITY_NODE,hour,2023-09-01T17:00:00Z,2023-09-01T18:00:00Z,37.878635,-122.301728,,,,,,,,,,,,,0.71,12.45,1,0.91,13.27,,0.0,0.85,,,,,,0.0,9.67,0.02,0,0,0.88,13.18,,54.68,,21.33,,
DDWLW0383,SF3FLYVL,CLARITY_NODE,hour,2023-09-01T17:00:00Z,2023-09-01T18:00:00Z,19.746,-155.087,,,,,9.23,9,9,,,,,,7.27,7.27,7,8.76,8.76,,4.69,8.38,,,,,,7.08,7.08,7.08,30,7,8.72,8.72,,45.12,,21.81,,
DEFSL2364,RW1075XQ,REFERENCE_SITE,hour,2023-09-01T17:00:00Z,2023-09-01T18:00:00Z,45.4369,-75.7261,,,,,,,,31.0,31,,,,,,,,,,,,6.4,,,,6,6.4,,,,6,,,,,,,,
...more...

Example with qcAssessment and qcFlags:

"datasourceId","sourceId","sourceType","outputFrequency","startOfPeriod","endOfPeriod","locationLatitude","locationLongitude","pm2_5ConcMass1HourMean.qcAssessment","pm2_5ConcMass1HourMean.qcFlags","pm2_5ConcMass1HourMean.raw","pm2_5ConcMass1HourMean.status","pm2_5ConcMass1HourMean.value"
DNIXT8160,AVWPGWWW,CLARITY_NODE,hour,2024-10-07T17:00:00Z,2024-10-07T18:00:00Z,37.87902695401437,-122.30192899608234,invalid,"[""QC.I.OOB.001"",""QC.I.OOB.002""]",5896.15,calibrated-ready,25.75

Example code

Slicing specific columns from CSV

The following sample Python code selects just the columns you want and converts to native Python types.

# simple demo using Clarity Data API

import requests
import os
import csv
import pprint
import datetime
import json


BASEURL = 'https://clarity-data-api.clarity.io'
HEADERS = {
    'Accept-Encoding': 'gzip',
    'x-api-key': os.environ.get('MY_CLARITY_API_KEY') # put your key in the environment or directly here
}
ORG_ID = '<your org id goes here>'  # you can find it in Dashboard under the menu item Organization>Resources
DATASOURCE_ID = '<the id of a datasource in your org>'


def check_can_connect():
    # verify can reach the API
    response = requests.get(BASEURL, HEADERS)
    http_code = response.status_code
    connected = http_code == 200
    if connected:
        print("Connected to Clarity")
    else:
        print(f"{http_code}  :(  Cannot connect")


def get_recent_measurements(
    org, datasource_ids, output_frequency, format, metric_selector
):
    # Fetch measurements from the API
    url = BASEURL + "/v2/recent-datasource-measurements-query"
    request_body = {
        "org": org,
        "datasourceIds": datasource_ids,
        "outputFrequency": output_frequency,
        "format": format,
        "metricSelect": metric_selector,
        "qcAssessment": True,
        "qcFlags": True,
    }
    response = requests.post(url, headers=HEADERS, json=request_body)
    response.raise_for_status()
    return response.text


def csv_to_typed(csv_data, fields_to_extract):
    # slice specific columns out of the CSV-style response
    # convert each to best Python type
    # return dictionary

    def convert_to_best_type(value, name):
        # Try these types in order: float, datetime, else string
        if value and ("qcFlags" in name or "Array" in name or "Vector" in name):
            try:
                return json.loads(value)
            except:
                pass
        try:
            return float(value)
        except Exception:
            try:
                return datetime.datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ")
            except Exception:
                return value

    reader = csv.DictReader(csv_data.splitlines())
    return [
        {field: convert_to_best_type(row[field], field) for field in fields_to_extract}
        for row in reader
    ]


check_can_connect()

tabular = get_recent_measurements(
    org=ORG_ID,
    datasource_ids=[DATASOURCE_ID],
    format="csv-wide",
    output_frequency="hour",
    metric_selector="only pm2_5ConcMass1HourMean",
)

measurements = csv_to_typed(
    tabular,
    [
        "startOfPeriod",
        "pm2_5ConcMass1HourMean.value",
        "pm2_5ConcMass1HourMean.status",
        "pm2_5ConcMass1HourMean.qcAssessment",
        "pm2_5ConcMass1HourMean.qcFlags",
    ],
)

pprint.pprint(measurements)