> ## Documentation Index
> Fetch the complete documentation index at: https://docs.adaptive-ml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Threads

> The core data structure for training and inference in Harmony recipes

`StringThread` is the atomic data element in `adaptive_harmony`. A thread is a sequence of turns (role + content), combined with turn weights for training and optional metadata (metric feedback, ground truth labels, or any custom key-value pairs).

## Create a StringThread

```python theme={null}
from adaptive_harmony import StringThread

thread = StringThread(
    turns=[
        ("user", "Hello, who are you?"),
        ("assistant", "I am a large language model. How can I help you today?"),
    ]
)

thread_with_metadata = StringThread(
    turns=[
        ("user", "Hello, who are you?"),
        ("assistant", "I am a large language model. How can I help you today?"),
    ],
    metadata={"label": "polite"}
)
```

### Builder methods

Each method returns a new `StringThread` with the turn appended:

```python theme={null}
thread = StringThread([])
thread = thread.system("Your name is Adaptive.")
thread = thread.user("Hello there!")
thread = thread.assistant("Hello, I'm Adaptive, and here to help you!")
thread = thread.tool('{"result": "success"}')
```

## Access turns and content

```python theme={null}
# All turns as (role, str content) tuples
all_turns = thread.get_turns()

# All turns except the last assistant turn
messages = thread.messages()

# The last assistant turn's content, or None
completion = thread.completion()

# Content of the very last turn (any role)
last = thread.last_content()
```

`get_turns()` returns every turn as a `(role, content)` tuple. For multimodal turns, images are represented as `<|image|>` in the string content.

`messages()` returns all turns except the final one if it has the `assistant` role. This is useful when you need to split a thread into prompt and completion.

## Multimodal StringThread

The difference with text-only is that `content` becomes a list of fragments instead of a plain string. There are two fragment types:

| Type            | Content key | Example                                                 |
| --------------- | ----------- | ------------------------------------------------------- |
| `TextFragment`  | `text`      | `{"type": "text", "text": "Describe this image."}`      |
| `ImageFragment` | `url`       | `{"type": "image", "url": "data:image/png;base64,..."}` |

Use `StringThread.from_fragments()` to create a multimodal thread. Don't forget the `await`! `from_fragments` is async because it loads and decodes images.

```python theme={null}
from adaptive_harmony import StringThread, TextFragment, ImageFragment

thread = await StringThread.from_fragments([
    ("user", [
        TextFragment(text="What's in this image?"),
        ImageFragment(url=f"data:image/png;base64,{image_data}"),
    ]),
])
```

You can also pass fragments as plain dictionaries:

```python theme={null}
thread = await StringThread.from_fragments([
    ("user", [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image", "url": f"data:image/png;base64,{image_data}"},
    ]),
])
```

A text-only `StringThread` is equivalent to a fragment thread with a single `TextFragment`:

```python theme={null}
# These two are equivalent
thread = StringThread([("user", "Hello world!")])
thread = await StringThread.from_fragments([
    ("user", [{"type": "text", "text": "Hello world!"}])
])
```

<Warning>
  Only `user` and `system` roles can contain images. The `assistant` role must be text-only.
</Warning>

<Info>
  The fragment format in Harmony differs from the [chat completions API](/v0.14/core/inference). In Harmony, image fragments use `{"type": "image", "url": "..."}`, while the chat completions API uses `{"type": "image_url", "image_url": {"url": "..."}}`.
</Info>

### Image encoding

`ImageFragment` expects a `url` field with the full data URI. To base64-encode a local image, use the built-in helper:

```python theme={null}
from adaptive_harmony.core.image_utils import image_to_base64

b64 = image_to_base64("photo.png")
fragment = ImageFragment(url=f"data:image/png;base64,{b64}")
```

`image_to_base64` returns the raw base64 string (without the `data:...` prefix). It also allows you to resize images and convert to grayscale:

| Parameter               | Type        | Description                                |
| ----------------------- | ----------- | ------------------------------------------ |
| `image_path`            | str or Path | Path to the image file                     |
| `format`                | str         | Output format (default: `"PNG"`)           |
| `longest_side_max_size` | int or None | Resize so the longest side fits this limit |
| `black_and_white`       | bool        | Convert to grayscale (default: False)      |

### Supported image formats

The formats accepted depend on the context:

* **In recipes** (`adaptive_harmony`): most image formats are supported (PNG, JPEG, GIF, WebP, BMP, TIFF, etc.). Images can be loaded from file paths, HTTP URLs, or `data:` URIs.
* **Via the chat completions API** (SDK / OpenAI client): only **PNG, JPEG, GIF, and WebP** are accepted, and only `data:` URIs, HTTP URLs are rejected.

We recommend using **PNG or JPEG** for maximum compatibility.

## Turn weighting

During training, turn weights control how much each turn contributes to the loss. A weight of `0.0` means the model does not learn from that turn, while `1.0` means it contributes fully. This is how you tell the model which parts of a conversation to learn from: typically you want the model to learn from assistant responses, not from user prompts or system messages.

By default, turns added with `.assistant()` get a weight of `1.0` and all other roles get `0.0`. When you [load a dataset](/v0.14/harmony/datasets) that contains completions, `with_weight_last_assistant_turn()` is applied automatically: only the final assistant turn is weighted. You can override this after loading using one of the methods below.

### Weighting methods

Each method returns a new `StringThread` with updated weights:

| Method                                      | Behavior                                                            |
| ------------------------------------------- | ------------------------------------------------------------------- |
| `with_weight(w)`                            | Set weight `w` on all turns                                         |
| `with_weight_all_assistant_turns()`         | Weight 1.0 on all assistant turns, 0.0 on others                    |
| `with_weight_last_assistant_turn()`         | Weight 1.0 on the last assistant turn only, 0.0 on others           |
| `with_weight_assistant_turns_from_index(i)` | Weight 1.0 on assistant turns starting from the i-th assistant turn |

```python theme={null}
# Train on all assistant responses in a multi-turn conversation
thread = thread.with_weight_all_assistant_turns()

# Train only on the final assistant response
thread = thread.with_weight_last_assistant_turn()

# Train on assistant turns starting from the 2nd one (index 1)
thread = thread.with_weight_assistant_turns_from_index(1)
```

```python theme={null}
# Override default weighting after loading a dataset
dataset = await config.dataset.load(ctx)
dataset = [thread.with_weight_all_assistant_turns() for thread in dataset]
```

### Inspect weights

```python theme={null}
# Returns list of weight for each turn
turns_with_weights = thread.get_turn_weights()
```