# Table Header Consolidation Algorithm

## Problem Statement

When processing documents with Docling, tables often have multiple consecutive headers at the same level (e.g., multiple H2 headers) immediately preceding them. Current implementations may arbitrarily select only the last header, losing important context. For example:

```
## Financial & Operational Tables
## Consolidated Statements of Comprehensive Income
## T-Mobile US, Inc. (Unaudited)
[TABLE]
```

Without consolidation, only "(Unaudited)" might be captured as the table title, missing critical context.

## Algorithm Overview

The Table Header Consolidation Algorithm identifies and consolidates truly consecutive headers of the same level that immediately precede tables in a document, combining them with a separator to preserve full context.

## Core Rules

### 1. Strict Consecutiveness
Headers must be **absolutely consecutive** in the document flow with zero intervening elements. Any element between headers breaks the sequence.

### 2. Same-Level Only
Only consolidate headers at the exact same level (all H1, all H2, etc.). Mixed levels are never consolidated.

### 3. Text Elements Only
Only process text elements with header markup. Picture elements are never consolidated, even if they appear at the same document level.

### 4. Content Barriers
The following elements immediately break a header sequence:
- Picture/Image elements
- Regular text paragraphs
- Tables
- Lists
- Any non-header element
- Headers of different levels

### 5. Table-Specific Application
Consolidation is ONLY applied when processing headers that immediately precede tables. Headers elsewhere in the document are not modified.

### 6. Backward Collection
Headers are collected by walking backward from the table position until a non-qualifying element is encountered.

## Algorithm Implementation

### Input Parameters
- `document`: Docling document object
- `separator`: String to join headers (default: " ^ ")
- `max_lookback`: Maximum elements to examine backward (default: 10)
- `consolidate_same_level`: Boolean to enable same-level consolidation (default: True)

### Process Flow

```python
function consolidate_headers_before_table(document, table_index):
    headers = []
    current_index = table_index - 1
    header_level = None

    while current_index >= 0 and (table_index - current_index) <= max_lookback:
        element = document.body.children[current_index]
        resolved_element = element.resolve(document)

        # Check if element is a text header
        if not is_text_element(resolved_element):
            break  # Non-text element breaks sequence

        if not is_header_element(resolved_element):
            break  # Non-header breaks sequence

        element_level = get_header_level(resolved_element)

        # First header sets the level
        if header_level is None:
            header_level = element_level
        elif header_level != element_level:
            break  # Different level breaks sequence

        # Add header to collection (in reverse order)
        headers.insert(0, resolved_element.text.strip())
        current_index -= 1

    # Consolidate if multiple headers found
    if len(headers) > 1:
        return separator.join(headers)
    elif len(headers) == 1:
        return headers[0]
    else:
        return None
```

### Helper Functions

#### `is_text_element(element)`
Returns `True` if element is a text element (not Picture, Table, etc.)

#### `is_header_element(element)`
Determines if a text element is a header by checking:
- Element label/type (e.g., "section_header", "heading")
- Text patterns (short, title-like content)
- Markdown indicators (starts with #)

#### `get_header_level(element)`
Extracts the header level (H1-H6) from element metadata or markdown syntax

## Examples

### Case 1: Valid Consolidation
```
## Financial Tables
## Income Statement
## (Unaudited)
[TABLE]
```
Result: `"Financial Tables ^ Income Statement ^ (Unaudited)"`

### Case 2: Image Breaks Sequence
```
## Financial Tables
[IMAGE]
## Income Statement
## (Unaudited)
[TABLE]
```
Result: `"(Unaudited)"` (only the last header)

### Case 3: Mixed Levels
```
# Main Section
## Financial Tables
## Income Statement
[TABLE]
```
Result: `"Financial Tables ^ Income Statement"` (H1 not included)

### Case 4: Intervening Text
```
## Financial Tables
## Income Statement
This table shows quarterly results.
[TABLE]
```
Result: `"Financial Tables ^ Income Statement"`, with
`"This table shows quarterly results."` captured separately as descriptive
text.

A single paragraph directly above the table does not break the sequence. The
walk stores the first such paragraph as descriptive text and keeps going, so
the headers above it still consolidate. A paragraph only breaks the sequence
once at least one header has already been collected, which is what makes
consecutive headers consecutive.

### Case 5: Images Between Headers
```
## Header 1
[IMAGE]
## Header 2
[IMAGE]
## Header 3
[TABLE]
```
Result: `"Header 3"` (images break all sequences)

## Configuration Options

### Flags
- `enable_consolidation`: Master toggle for feature (default: True)
- `separator`: String to join headers (default: " ^ ")
- `max_lookback`: Maximum elements to check backward (default: 10)
- `preserve_formatting`: Maintain original header formatting (default: False)
- `trim_headers`: Remove extra whitespace from headers (default: True)

### Output Format Options
- `include_level_prefix`: Add level indicator (e.g., "H2: ") (default: False)
- `wrap_consolidated`: Add delimiters around consolidated title (default: False)
- `max_title_length`: Truncate if consolidated title exceeds length (default: None)

## Edge Cases

1. **Empty Headers**: Skip headers with no text content
2. **Duplicate Headers**: Include all instances, even if text is identical
3. **Very Long Sequences**: Respect `max_lookback` limit to prevent excessive processing
4. **Unicode/Special Characters**: Preserve as-is in consolidated output
5. **Nested Headers**: Only consider top-level document structure

## Integration Notes

- Algorithm operates on Docling document object before any export/serialization
- Does not modify the original document structure (creates new metadata)
- Compatible with all Docling export formats (Markdown, HTML, etc.)
- Images are saved as separate files with references, not embedded

## Testing Checklist

- [ ] Multiple same-level headers consolidate correctly
- [ ] Images break header sequences
- [ ] Mixed header levels handled properly
- [ ] Text content breaks sequences
- [ ] Single headers pass through unchanged
- [ ] Empty document sections handled gracefully
- [ ] Maximum lookback limit respected
- [ ] Special characters preserved correctly