ListWords API - Filtering and Querying¶

Overview¶

The ListWords API uses a unified CEL (Common Expression Language) based filtering approach for maximum flexibility and consistency across all List APIs.

API Definition¶

message ListWordsRequest {
  string filter = 1;      // CEL expression for filtering
  string order_by = 2;    // Sorting specification
  common.v1.PaginationRequest pagination = 3;
}

Filter Fields¶

The following fields are available for filtering:

Field	Type	Operators	Description
`language`	string	`==`, `!=`	ISO 639-1 language code (e.g., "en", "fr")
`keyword`	string	`==`, `!=`	Searches within lemma AND inflected forms (partial match, case-insensitive)
`category`	string[]	`in`	Filters by word categories/tags (OR logic)
`surface`	string[]	`in`	Batch lookup by exact lemma OR inflected forms (exact match)

Filter Examples¶

Language Filtering¶

Find all English words:

filter: 'language == "en"'

Find all French words:

filter: 'language == "fr"'

Keyword Search¶

The keyword field searches in both the lemma and all inflected forms, making it ideal for fuzzy search:

Search for words containing "book":

filter: 'keyword == "book"'
// Finds: "book", "books", "booking", etc.

Search by inflected form - finds the base word:

filter: 'keyword == "apples"'
// Finds: "apple" (because "apples" is its plural form)

filter: 'keyword == "running"'
// Finds: "run" (because "running" is its present participle)

Partial match:

filter: 'keyword == "swim"'
// Finds: "swim" (matches lemma and forms like "swimming", "swam")

Category Filtering¶

Find CET-4 level words:

filter: 'category in ["cet4"]'

Find words that are in either CET-4 OR CET-6:

filter: 'category in ["cet4", "cet6"]'

Surface Term Lookup (Batch Lookup)¶

The surface field enables exact match batch lookup of words by their lemma or inflected forms.

Key differences from keyword: - keyword: Partial match (contains), searches in both lemma and forms - surface: Exact match, used for batch lookup of specific terms

The surface field is particularly useful for: - Looking up multiple specific words in one query - Finding words by exact inflected forms (e.g., "running" → "run") - Batch operations with exact term matching

Find word by exact lemma:

filter: 'surface in ["run"]'

Find word by exact inflected form:

filter: 'surface in ["running"]'
// Returns: "run" (exact match on the form "running")

Batch lookup multiple words:

filter: 'surface in ["run", "swim", "walk"]'
// Returns all three words

Batch lookup by mixed lemmas and forms:

filter: 'surface in ["running", "swam", "walked"]'
// Returns: run, swim, walk (by their inflected forms)

Combined Filtering¶

English words in technology category:

filter: 'language == "en" && category in ["technology"]'

CET-4 words containing "comp":

filter: 'keyword == "comp" && category in ["cet4"]'

English words matching specific surface forms:

filter: 'language == "en" && surface in ["running", "swimming"]'

Sorting¶

The order_by field supports:

Field	Description
`lemma`	Sort by lemma alphabetically
`created_at`	Sort by creation time
`updated_at`	Sort by last update time

Add desc for descending order:

order_by: "lemma"           # Ascending (A-Z)
order_by: "lemma desc"      # Descending (Z-A)
order_by: "created_at desc" # Newest first

Pagination¶

Standard pagination with page number and page size:

pagination: {
  page_no: 1,
  page_size: 20
}

page_no: Page number (1-indexed)
page_size: Number of items per page (max: 100)

Complete Examples¶

Example 1: English CET-4 words, sorted alphabetically¶

{
  "filter": "language == \"en\" && category in [\"cet4\"]",
  "order_by": "lemma",
  "pagination": {
    "page_no": 1,
    "page_size": 50
  }
}

Example 2: Batch lookup with sorting¶

{
  "filter": "surface in [\"running\", \"swimming\", \"walking\"]",
  "order_by": "lemma",
  "pagination": {
    "page_no": 1,
    "page_size": 10
  }
}

Example 3: Technology words updated recently¶

{
  "filter": "language == \"en\" && category in [\"technology\"]",
  "order_by": "updated_at desc",
  "pagination": {
    "page_no": 1,
    "page_size": 20
  }
}

Implementation Details¶

Surface Term Lookup¶

The surface field uses a sophisticated query that: 1. Matches the word's lemma directly (case-insensitive) 2. Joins with the lexemes and lexeme_forms tables to find words with matching inflected forms 3. Uses OR logic to return words that match ANY of the provided surface terms

SQL logic (simplified):

SELECT * FROM words 
WHERE 
  LOWER(lemma) IN ('running', 'swimming') OR
  EXISTS (
    SELECT 1 FROM lexemes l
    JOIN lexeme_forms f ON l.id = f.lexeme_id
    WHERE l.word_id = words.id 
      AND LOWER(f.text) IN ('running', 'swimming')
  )

This enables efficient batch lookup while maintaining the ability to find words by any of their forms.

Category Filtering Logic¶

Category filtering uses OR logic: - category in ["cet4", "cet6"] returns words that have EITHER "cet4" OR "cet6" in their categories array - A word with categories: ["cet4", "technology"] will match the above filter

Performance Considerations¶

Keyword searches use case-insensitive partial matching
Surface term lookups use indexed queries for efficiency
Category filtering leverages JSONB array operators in PostgreSQL
Pagination is applied after filtering and sorting to minimize data transfer