Clusters API¶

Per-commodity village clusters generated by the LEAF DSS clustering engine. A cluster is a small group of nearby villages within one block whose total interested members in a single commodity (e.g. dairy, goatery) fall inside the government-mandated band (default 30-150). One village can belong to multiple clusters - clusters are commodity-specific, not village-specific.

Algorithm

Greedy spatial seed-and-grow within a block. Each commodity is processed independently. Constraints (all tunable):

Total members in cluster: 30-150
Villages per cluster: any count, as long as the cluster fits within ≤ 5 km max pairwise span (LEAF-43 removed the prior 2-4 band)
Block boundary is a hard wall - clusters never cross blocks
Per-village minimum interest: 6 members in the commodity (LEAF-42: ≤5 excluded as "unassigned")

Defaults reflect government criteria captured during the IWMI requirements call (2026-04-23) plus the LEAF-42 / LEAF-43 changes from the 2026-05-21 call. Override per request via the body of POST /api/clusters/regenerate.

How Clustering Works¶

Plain-English walkthrough of one (block, commodity) run - implemented in leaf_flask/clustering.py.

Step-by-step¶

Filter candidates. From all villages in the block, keep those with member count ≥ min_members_per_village for that commodity. Villages with zero interest are excluded.
Sort by interest, descending. The village with the largest member count for the commodity goes first - this becomes the next seed. Sorting big-first makes it more likely that a cluster can hit the 30-member floor before running out of nearby villages.
Grow the cluster greedily from the seed. Repeatedly find the nearest unassigned village to the seed that satisfies all three constraints:
1. Within max_radius_km of the seed (haversine distance).
2. Adding it keeps the cluster's max pairwise span ≤ max_radius_km - so when there are already several villages, the next must be close to every existing member, not only the seed.
3. Adding its members doesn't push the cluster total over max_cluster_members.
(LEAF-43 removed the prior 4-village ceiling — clusters of 5+ villages are valid as long as the member band and 5 km span still hold.) Add it, mark assigned, repeat until no candidate fits. 4. Validate. When growth stops, the cluster is kept only if it has ≥ min_cluster_members members. Otherwise it is discarded and its villages are released back to the unassigned pool so they can seed/join later clusters. A single-village cluster with members in the funding band is valid. 5. Persist. Compute centroid (mean lat/lon), max pairwise span, and assign a UUID-suffixed cluster ID like KHOWANG-Goatery-0267432b. Store in Postgres. 6. Next seed. Move to the next biggest unassigned village and repeat from step 3. Stop when no candidates remain. 7. Next commodity. Run the whole thing again for each of the six commodities, independently against the full village list.

Why one village can appear in many clusters¶

Each commodity is clustered independently. The same village can sit in a Goatery cluster and a Piggery cluster and a Duckery cluster simultaneously - because its goat-interested members, pig-interested members, and duck-interested members are different sub-populations. This matches the IWMI design intent: clusters are commodity-specific, not village-specific.

Why greedy seed-and-grow (vs. K-means / DBSCAN)¶

Approach	Why it doesn't fit
K-means	Doesn't respect hard size bands (30-150 members). You'd have to post-filter, throwing away most clusters.
DBSCAN	Density-based blobs, no bounded cluster size; can produce one giant cluster covering the whole block.
Greedy seed-and-grow	Bakes in every IWMI constraint (radius, size band, block boundary) at the construction step. Simpler, deterministic given a seed order, and trivially explainable to non-technical reviewers.

Block boundary as a hard wall¶

Per the IWMI requirements call: "my algorithm external boundary is my block boundary." The village DataFrame is filtered to a single block before clustering runs, so there is no path for a cluster to span two blocks.

Tuning¶

Every constraint is a request-time parameter - pass them in the body of POST /api/clusters/regenerate. Examples:

max_radius_km: 3 → tighter, fewer clusters.
min_cluster_members: 20 → more clusters by relaxing the floor.
max_cluster_members: 200 → larger clusters allowed.

Worked example¶

KHOWANG-Goatery-0267432b, 44 total members:

Seed = the largest unassigned goat-interested village in KHOWANG (say ~30 members).
Nearest neighbour within 5 km that has ~14 members joins → cluster total 44.
Tries a third village - every candidate either pushes the cluster over 50 members or is more than 5 km from one of the existing two. None fits.
Cluster has 44 members and 2 villages, both ≥ floors → kept.

Get Default Parameters¶

GET /api/clusters/params

Response¶

{
  "min_members_per_village": 6,
  "min_cluster_members": 30,
  "max_cluster_members": 150,
  "max_radius_km": 5.0,
  "emit_provisional": true,
  "provisional_min_members": 1,
  "rebalance": false,
  "commodities": ["Dairy", "Goatery", "Piggery", "Backyard_Poultry", "Duckery", "Fishery_Activity"]
}

Param	Type	Description
`min_members_per_village`	integer	Villages with fewer than this many interested members for the commodity are filtered out before clustering and hidden from the cluster map. Default `6` (LEAF-42: ≤5 excluded).
`emit_provisional`	boolean	When `true`, villages that never reach `min_cluster_members` are surfaced as provisional clusters (relaxed floor, flagged `provisional: true`) instead of being dropped. Default `true`.
`provisional_min_members`	integer	Member floor for provisional clusters when `emit_provisional` is on. Default `1` (surface everything).
`rebalance`	boolean	When `true`, a below-floor provisional group may become fundable by borrowing village(s) from an adjacent fundable cluster — but only when every cap still holds (≤ `max_radius_km` span, ≤ `max_cluster_members`) and the donor stays valid (≥ `min_cluster_members`). Conservative by design: takes the smallest borrow that clears the floor, processes provisional groups largest-first, moves each village at most once, and reverts a group's borrows entirely if it can't reach the floor. Default `false` — enabling it changes output broadly, so turn it on deliberately. LEAF-43 removed the village-count cap so Pass C handles most rescues now; Pass E mostly fires when Pass C is blocked by the member ceiling.

List Clusters¶

GET /api/clusters

Returns stored clusters, optionally filtered.

Smart auto-refresh (no manual Regenerate needed)

When a block is supplied, each in-scope (block, commodity) is transparently regenerated only when it is stale — the algorithm version, params, or underlying village data changed since it was last generated — and the scope is not locked (no finalized clusters, no uploaded-CSV edits). Fresh scopes and human-owned scopes are returned untouched, so reloads never wipe edits and unchanged scopes aren't needlessly rebuilt. Without a block, no regeneration happens (avoids a full-corpus run). The clustering params above can be passed as query params to scope the staleness check.

Query Parameters¶

Param	Type	Required	Description
`block`	string	No	Filter by `block_name`. Also enables smart auto-refresh for the scope.
`district`	string	No	Filter by `district_name`.
`commodity`	string	No	One of `Dairy`, `Goatery`, `Piggery`, `Backyard_Poultry`, `Duckery`, `Fishery_Activity`.

Response¶

[
  {
    "cluster_id": "KHOWANG-Goatery-d11142b8",
    "cluster_code": "DI-KH-GO-01",
    "cluster_name": null,
    "commodity": "Goatery",
    "block_name": "KHOWANG",
    "district_name": "DIBRUGARH",
    "total_members": 49,
    "max_span_km": 1.193,
    "centroid_lat": 27.204413,
    "centroid_lon": 94.80894,
    "villages": [
      { "vill_name": "KHELMATI",     "gp_name": "DIKHARI TILOI", "lat": 27.20, "long": 94.80, "members": 47 },
      { "vill_name": "NO.3 LENGERI", "gp_name": "LENGERI",       "lat": 27.21, "long": 94.81, "members": 2  }
    ],
    "pashu_sakhi": null,
    "block_coordinator": null,
    "district_coordinator": null,
    "finalized": false,
    "locked": false,
    "provisional": false
  }
]

Field	Type	Description
`cluster_num`	integer	Display sequence number within its tier, restarting per (block, commodity). Fundable and provisional groups are numbered in separate sequences. Display-only, not persisted.
`cluster_label`	string	Display label built from `cluster_num`: `"1"`, `"2"`, … for fundable clusters and `"P1"`, `"P2"`, … for provisional groups, so a review-only group never reads as a peer of a fundable cluster.
`cluster_code`	string	Human-readable unique cluster ID shown throughout the UI: hyphen-separated first two letters of district + block + commodity (intervention type) followed by the two-digit sequence number — e.g. `MO-BH-GO-01` (MOrigaon / BHurbandha / GOatery / fundable #1). Provisional groups carry a `P` before the number: `MO-BH-GO-P01`. Derived at read time from `cluster_num`, so CSV splits/merges renumber on the next read; `cluster_id` stays the stable key.
`cluster_name`	string	Optional editable display-name override for `cluster_code`. When set (via the CSV edit/import cycle) the UI shows it instead of the auto code everywhere the cluster name appears — the filter-bar dropdown, the cluster ID ribbon and the block cluster cards. `null`/empty keeps the auto `cluster_code`.
`district_coordinator`	string	Optional District Coordinator name (DC), shown in the Contact Persons card. `null` until assigned via the CSV edit/import cycle. Sits alongside `block_coordinator` (BC) and `pashu_sakhi` (PS).
`provisional`	boolean	`true` for below-floor clusters surfaced by `emit_provisional` — flagged for review, not fundable.
`locked`	boolean	`true` when the cluster is human-owned (uploaded via CSV or finalized). Locked scopes are never auto-regenerated.
`finalized`	boolean	`true` once published to the Production Tool feed.

Get Cluster by ID¶

GET /api/clusters/<cluster_id>

Returns one cluster by its internal cluster_id. The record carries the same derived cluster_num / cluster_label / cluster_code fields as the list endpoint.

Code	Description
`200`	Cluster record.
`404`	Cluster ID not found in store.

Cluster Report Card¶

GET /api/clusters/<cluster_id>/report

Cluster dashboard view. Returns the cluster plus all variables for the parent block (soil, water, climate, infrastructure, people, livestock, land/agri) - pulled from the existing LEAF block dataset. Mirrors slide 7 of the LEAF DSS Clustering Workflow deck.

Response¶

{
  "cluster": { "...": "as in /api/clusters/<id>" },
  "block": {
    "Block_name": "KHOWANG",
    "Dist_Name": "DIBRUGARH",
    "AD": 42.1,
    "WA": 18.5,
    "...": "all LEAF variables for the parent block"
  }
}

Code	Description
`200`	Always returned when the cluster exists; `block` is `{}` if the parent block isn't in the LEAF shapefile.
`404`	Cluster not found.

Regenerate Clusters¶

POST /api/clusters/regenerate

Runs the algorithm and replaces stored clusters within scope. Scope precedence: (block + commodity) > (block) > all blocks / all commodities.

Admin-only endpoint

Pass ?admin=1 or header X-Admin: 1. The endpoint returns 403 without it. Regenerate wipes any CSV edits in scope, so it is hidden from the UI for regular users and gated server-side as defense-in-depth (per Faiz, 2026-05-09 call).

Request Body¶

All fields optional; missing parameters fall back to /api/clusters/params defaults.

{
  "block": "KHOWANG",
  "commodity": "Goatery",
  "min_cluster_members": 30,
  "max_cluster_members": 150,
  "max_radius_km": 5.0,
  "min_members_per_village": 6
}

Response¶

{
  "count": 81,
  "clusters": [ /* same shape as GET /api/clusters */ ]
}

Example¶

cURLPython

curl -X POST "https://leaf-asrlm.in/api/clusters/regenerate?admin=1" \
     -H "Content-Type: application/json" \
     -H "X-Admin: 1" \
     -d '{"block": "KHOWANG", "commodity": "Goatery"}'

import requests
r = requests.post(
    "https://leaf-asrlm.in/api/clusters/regenerate",
    params={"admin": "1"},
    headers={"X-Admin": "1"},
    json={"block": "KHOWANG", "max_radius_km": 4.0},
)
print(r.json()["count"], "clusters")

Export Clusters as CSV¶

GET /api/clusters/export.csv

Returns a row-per-village CSV (one row for each village inside each cluster). Same scope filters as /api/clusters.

Query Parameters¶

Param	Type	Required
`block`	string	No
`district`	string	No
`commodity`	string	No

CSV Columns¶

Schema (LEAF-44, 2026-05-29): lat / long are emitted alongside vill_name so a user can append a row for a brand-new field-surveyed village by typing in its coordinates inline.

Column	Description
`cluster_code`	Human-readable auto code (e.g. `MO-BH-GO-01`; provisional `MO-BH-GO-P01`). Type a name over it to rename the cluster — on upload an edited code (one that differs from the auto value) is saved as `cluster_name`. Leave it unchanged to keep the auto code.
`cluster_name`	Optional editable display-name override for `cluster_code` (second column). Fill in to give the cluster a custom name shown throughout the UI; a blank cell keeps the auto code. Wins over an edited `cluster_code` when both are set.
`cluster_num`	The cluster's number (`1`, `2`, … / `P1`, `P2`, … for the provisional tier) within the `(block, commodity)` scope. This is the grouping key on upload: change a village's number to move it to that cluster, give rows the same number to merge, or a new number to split. Each existing number maps back to the cluster that currently holds it (stable `cluster_id`); a new number mints a new cluster; the `P` prefix marks the provisional tier.
`cluster_id`	Internal server-assigned ID. No longer required from editors — leave as-is; grouping follows `cluster_num`. Used only as a fallback for rows that have no number.
`commodity`	One of the six commodities.
`district_name`, `block_name`	Parent district / block.
`gp_name`, `vill_name`	Village identification.
`lat`, `long`	Village coordinates. For known villages these round-trip from the master. For a new village added via the CSV, supply the coordinates here.
`members`	Members in this village interested in the commodity.
`pashu_sakhi`	Optional - fill in to assign a Pashu Sakhi to the cluster.
`block_coordinator`	Optional - fill in to assign a block coordinator.
`district_coordinator`	Optional - last column. Fill in to assign a District Coordinator (shown in the Contact Persons card's DC row). Omitted from CSVs exported before this column existed; those still import with the DC left empty.

Grouping on upload follows cluster_num, not cluster_id (users edit the friendly number): move a village by changing its number to the destination cluster's number, merge by giving rows the same number, split by giving rows a new number, and rename by typing over the cluster_code column. Each existing number is mapped back to the cluster that currently holds it so the internal id stays stable; a number not yet in use mints a new cluster, and a P-prefixed number stays in the provisional tier.

Legacy CSVs without lat/long columns are still accepted: the parser backfills from the village master by (block_name, vill_name). An unknown vill_name then fails fast with a message pointing at the row. CSVs that predate the district_coordinator column are likewise accepted, with the DC field left empty.

Refresh All Clusters (whole-state coverage)¶

POST /api/clusters/refresh-all?admin=1

Rebuilds clusters for every (block, commodity) that is stale and not locked, so the whole-state export/report covers all 220 blocks — not just the ones a user opened on the map.

Why this exists

Clusters are generated lazily, per block, the first time someone opens that block. The whole-state export (export.csv with no block) only returns what is already stored, so any block nobody opened contributes zero, and the export can show as little as ~40% of the true member total (Faiz, 2026-07-02). This endpoint closes that gap; a daily scheduled job runs the same routine automatically.

Asynchronous. Returns immediately (202) with started: true and the current coverage; a full rebuild runs in the background. Poll GET /api/clusters/coverage to watch assigned climb.
Safe to re-run. Finalised/edited (locked) scopes are skipped so human edits survive; fresh scopes are skipped by fingerprint (a run with no data change does almost nothing); an advisory lock prevents overlapping runs.
Admin-only. Requires ?admin=1 or X-Admin: 1.

Example Response (`202`)¶

{
  "started": true,
  "coverage": [
    { "commodity": "Goatery", "raw_total_members": 1006945, "assigned_members": 423551, "unassigned_members": 583394, "assigned_pct": 42.1, "blocks_with_clusters": 86, "total_blocks": 220 }
  ]
}

Coverage Summary¶

GET /api/clusters/coverage

Per-commodity reconciliation for the whole state: raw interested members (from the village master) vs members currently assigned to stored clusters vs the unassigned remainder. Lets a reviewer confirm assigned + unassigned = total and spot coverage gaps. Reflects whatever is currently stored — run POST /api/clusters/refresh-all first (or wait for the daily job) for full-coverage numbers.

Response Fields (array, one per commodity)¶

Field	Description
`commodity`	One of the six commodities.
`raw_total_members`	Total interested members across all villages (village master).
`interested_villages`	Villages with ≥1 interested member.
`assigned_members`	Members currently in stored clusters.
`unassigned_members`	`raw_total_members − assigned_members`.
`assigned_pct`	Assigned as a percentage of the raw total.
`clusters`	Stored cluster count.
`blocks_with_clusters` / `total_blocks`	Coverage: blocks with clusters out of 220. `total_blocks` is computed live from the village master (distinct `block_name` count), so it tracks the true block count rather than a hard-coded constant.

Export Unassigned Villages as CSV¶

GET /api/clusters/unassigned.csv[?commodity=<commodity>]

One row per village that shows interest in a commodity (≥1 interested member) but is not part of any stored cluster for it. Companion to export.csv so assigned + unassigned = total.

Query Parameters¶

Param	Type	Required	Description
`commodity`	string	No	Restrict to one commodity; omit for all six.

CSV Columns¶

Column	Description
`commodity`	The commodity the village is unassigned under.
`district_name`, `block_name`, `gp_name`, `vill_name`	Village identification.
`members`	Interested members in this village for the commodity.
`reason`	`below_min_per_village` (fewer than 6 interested members — hidden by design, LEAF-42) or `not_clustered` (≥6 but no cluster holds it — usually a block not yet materialised).

After a full refresh, almost every row is below_min_per_village; any not_clustered row flags a real coverage gap.

Pending Rename Reconciliation¶

GET /api/clusters/pending-renames

Lists blocks that have stored clusters but are not in the current village master — i.e. clusters sitting under an old/renamed block name. These blocks are excluded from /api/clusters/coverage (so it can never exceed 100%) and surfaced here instead, so a reviewer can migrate the cadre-owned clusters onto the new block name before they are otherwise dropped.

Returns an array sorted by member count descending. An empty array means every stored block still matches the current master.

Response¶

[
  {
    "block_name": "LAKHIPUR",
    "clusters": 12,
    "cadre_clusters": 3,
    "members": 4821
  }
]

Field	Type	Description
`block_name`	string	The old/renamed block name still holding stored clusters.
`clusters`	integer	Total clusters stored under this old name.
`cadre_clusters`	integer	Of those, how many are finalized, locked, or carry dashboard data — these must be migrated onto the new name during rename reconciliation, never dropped.
`members`	integer	Total interested members across those clusters.

Code	Description
`200`	Array of old-name blocks with counts (empty when none pending).
`500`	Server error.

Import Clusters from CSV¶

POST /api/clusters/import?block=<block>[&commodity=<commodity>]

Replaces stored clusters in the given scope with the uploaded CSV. block is required. Same edit-via-CSV pattern as the existing LEAF data update flow - re-upload as many times as needed. Derived fields (total_members, max_span_km, centroid_*) are recomputed automatically.

Body¶

Either:

multipart/form-data with a file field, or
Raw CSV in the request body with Content-Type: text/csv.

Response¶

{ "imported": 81 }

Code	Description
`200`	Number of clusters stored (`{ "imported": <n> }`).
`400`	Missing `block` query param, or no CSV content received.
`500`	Parse/store error (e.g. an unknown `vill_name` with no `lat`/`long` to backfill).

Edit Flow¶

GET /api/clusters/export.csv?block=KHOWANG&commodity=Goatery → download.
Open in Excel; merge / split / drop clusters; fill pashu_sakhi, block_coordinator.
POST /api/clusters/import?block=KHOWANG&commodity=Goatery → upload edited CSV.
When stable, POST /api/clusters/<cluster_id>/finalize to publish the cluster to the Production Tool feed.

Clusters API¶

How Clustering Works¶

Step-by-step¶

Why one village can appear in many clusters¶

Why greedy seed-and-grow (vs. K-means / DBSCAN)¶

Block boundary as a hard wall¶

Tuning¶

Worked example¶

Get Default Parameters¶

Response¶

List Clusters¶

Query Parameters¶

Response¶

Get Cluster by ID¶

Cluster Report Card¶

Response¶

Regenerate Clusters¶

Request Body¶

Response¶

Example¶

Export Clusters as CSV¶

Query Parameters¶

CSV Columns¶

Refresh All Clusters (whole-state coverage)¶

Example Response (202)¶

Coverage Summary¶

Response Fields (array, one per commodity)¶

Export Unassigned Villages as CSV¶

Query Parameters¶

CSV Columns¶

Pending Rename Reconciliation¶

Response¶

Import Clusters from CSV¶

Body¶

Response¶

Edit Flow¶

Example Response (`202`)¶