Handle ingestion failures

When sending usage events to Zenskar via API, it's important to understand how ingestion failures are handled, and how you can prevent data loss due to validation errors or network issues.

Why this matters

If a usage event does not show up in Zenskar, the assumption is often that it was lost by our system. In reality, ingestion can fail in two main ways:

Recommended approach: implement a dead letter queue (DLQ)

🔖

Definition

A dead letter queue (DLQ) is a mechanism to capture failed events so that they can be retried or examined later.

A DLQ is a holding area for failed events, either due to validation errors or delivery issues. This ensures no event is lost, even if initial ingestion fails.

sequenceDiagram
    participant Your system
    participant Zenskar API
    participant DLQ

    Your system->>Zenskar API: Send usage event
    alt Success
        Zenskar API-->>Your system: 200 OK
    else Validation error
        Zenskar API-->>Your system: 4xx error
        Your system->>DLQ: Store failed event
    else Network error
        Note right of Your system: No response received
        Your system->>DLQ: Store failed event
    end

Ways to implement a DLQ

You can implement a DLQ in many ways, depending on your system architecture and event volume. Here are some common approaches:

Error responses from Zenskar

Below is a sample of a valid event upload payload. Use this as a reference to ensure your requests are correctly structured:

[
  {
    "data": {
      "campaign_id": "sample campaign_id 8",
      "impressions": 74
    },
    "timestamp": "2025-06-28 23:44:47",
    "customer_id": "c03"
  }
]

When uploading usage events via the ingestion API, your request may be rejected with a 4xx HTTP status code if there are issues with the payload. This guide outlines all possible 4xx errors, explains their causes, and provides real examples to help you debug and resolve issues effectively.

HTTP 4xx error codes

{
  "error": "invalid character '}' looking for beginning of object key string"
}

{
  "error": "Payload too large"
}

{
  "error": "Invalid type for key: impressions. Expected Int64, got string"
}

Validation error messages

The API validates your event payload against a schema. If validation fails, you will receive a 422 response with a descriptive error message. Below are the types of validation errors you may encounter:

{
  "error": "Missing key: impressions"
}

  {
    "error": "Unexpected key in payload: extra_field"
  }

{
  "error": "Invalid type for key: campaign_id. Expected String, got float64"
}

  {
    "error": "Invalid type for key: impressions. Expected Int64, got string"
  }

  {
    "error": "Invalid type for key: value. Expected Float64, got string"
  }

  {
    "error": "Invalid type for key: is_active. Expected Bool, got string"
  }

{
  "error": "Invalid type for key: start_date. Expected Date32, got string"
}

  {
    "error": "Invalid type for key: timestamp. Expected Date32/DateTime64, got string"
  }

  {
    "error": "Invalid type for key: user_id. Expected UUID, got string"
  }

{
  "error": "Invalid type for key: data. Expected Object, got string"
}

  {
    "error": "Invalid type for key: nested_field. Expected Int64, got string"
  }

Example error responses

Here are some real-world examples of error responses you might see:

{
  "data": {
    "campaign_id": "sample_campaign_id_8",
    "impressions": "74"
  },
  "timestamp": "2025-06-28 23:44:47",
  "customer_id": "c03"
}

{
  "error": "Invalid type for key: impressions. Expected Int64, got string"
}

{
  "data": {
    "campaign_id": "sample_campaign_id_8"
  },
  "timestamp": "2025-06-28 23:44:47",
  "customer_id": "c03"
}

{
  "error": "Missing key: impressions"
}

{
  "data": {
    "campaign_id": "sample_campaign_id_8",
    "impressions": 74,
    "extra_field": "not_allowed"
  },
  "timestamp": "2025-06-28 23:44:47",
  "customer_id": "c03"
}

{
  "error": "Unexpected key in payload: extra_field"
}

Best Practices

• Check the schema: Make sure your payload matches the expected schema for the usage event.
• Use correct types: Numbers should be numbers, not strings; dates should be in the correct format.
• Avoid extra fields: Only include fields defined in the schema.
• Keep payloads small: Stay under 1MB per request.

Summary Table