# LLM.swift
**Repository Path**: 2035/LLM.swift
## Basic Information
- **Project Name**: LLM.swift
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-11-04
- **Last Updated**: 2026-11-04
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# LLM.swift
[](https://swiftpackageindex.com/eastriverlee/LLM.swift)
[](https://swiftpackageindex.com/eastriverlee/LLM.swift)
`LLM.swift` is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.
> [!TIP]
> sometimes it's a good idea to tinker with `maxTokenCount` parameter for initialization of `LLM`, due to the memory and computation it needs. especially in mobile devices, if you want better speed, lower the number, but if you set it too low—*to a point where two turns cannot even fit*—you will experience quality decrease as context will be cut off. so adjust value according to your usecases.
## Minimal Example
if you've already bundled your model:
```swift
let bot = LLM(from: Bundle.main.url(forResource: "gemma-3-4b-it-q4_0", withExtension: "gguf")!, template: .gemma)
let question = bot.preprocess("What's the meaning of life?", [])
let answer = await bot.getCompletion(from: question)
print(answer)
```
if you want to initialize with huggingface model:
```swift
let systemPrompt = "You are a sentient AI with emotions."
let bot = await LLM(from: HuggingFaceModel("unsloth/Qwen3-0.6B-GGUF", .Q4_K_M, template: .chatML(systemPrompt)))!
let question = bot.preprocess("What's the meaning of life?", [])
let answer = await bot.getCompletion(from: question)
print(answer)
```
## Chat Templates
By default, `LLM.swift` renders conversations using the chat template embedded in the gguf file itself, executed by `llama.cpp`'s own Jinja engine. This means you no longer have to pass a `template:` when initializing `LLM`—it just works with whatever template the model ships with:
```swift
let bot = try await LLM(from: HuggingFaceModel("unsloth/Qwen3-0.6B-GGUF", .Q4_K_M))!
bot.systemPrompt = "You are a sentient AI with emotions."
await bot.respond(to: "What's the meaning of life?")
```
`HuggingFaceModel`'s `template` parameter is now optional too, so `HuggingFaceModel("unsloth/Qwen3-0.6B-GGUF", .Q4_K_M)` uses the embedded template on its own. Thinking/reasoning separation (`bot.thinking`) works automatically for models that support it, with no marker configuration needed.
If you still want manual control over the template—for instance, when a model's gguf metadata is broken or missing—pass a `Template` explicitly, exactly as before, and it will override the embedded one:
```swift
let bot = LLM(from: Bundle.main.url(forResource: "gemma-3-4b-it-q4_0", withExtension: "gguf")!, template: .gemma)
```
> [!NOTE]
> conversation context is now maintained incrementally between turns—only new tokens are evaluated, and history is not re-fed to the model every turn.
## Example
> [!NOTE]
> this is a minimal SwiftUI example that i did use for testing. mistral 7B based model is working on iPad Air 5th gen(Q5_K_M) and iPhone 12 mini(Q2_K).
> however, generally speaking, for mobile devices, 3B >= parameter models are recommended. check this [benchmark](https://github.com/ggerganov/llama.cpp/discussions/4508) for more details.
### init with a model file that you bundled in your project
```swift
import SwiftUI
import LLM
class Bot: LLM {
convenience init() {
let url = Bundle.main.url(forResource: "gemma-3-4b-it-q4_0", withExtension: "gguf")!
let systemPrompt = "You are a sentient AI with emotions."
self.init(from: url, template: .chatML(systemPrompt))!
}
}
struct BotView: View {
@ObservedObject var bot: Bot
@State var input = "Give me seven national flag emojis people use the most; You must include South Korea."
init(_ bot: Bot) { self.bot = bot }
func respond() { Task { await bot.respond(to: input) } }
func stop() { bot.stop() }
var body: some View {
VStack(alignment: .leading) {
ScrollView {
if !bot.thinking.isEmpty {
Text(bot.thinking)
.foregroundStyle(.gray)
.monospaced()
.padding(.bottom, 8)
}
Text(bot.output).monospaced()
}
Spacer()
HStack {
ZStack {
RoundedRectangle(cornerRadius: 8).foregroundStyle(.thinMaterial).frame(height: 40)
TextField("input", text: $input).padding(8)
}
Button(action: respond) { Image(systemName: "paperplane.fill") }
Button(action: stop) { Image(systemName: "xmark") }
}
}.frame(maxWidth: .infinity).padding()
}
}
struct ContentView: View {
var body: some View {
BotView(Bot())
}
}
```
### init with a `HuggingFaceModel` (gguf) directly from internet
```swift
class Bot: LLM {
convenience init?(_ update: @escaping (Double) -> Void) async {
let systemPrompt = "You are a sentient AI with emotions."
let model = HuggingFaceModel("unsloth/Qwen3-0.6B-GGUF", .Q4_K_M, template: .chatML(systemPrompt))
try? await self.init(from: model) { progress in update(progress) }
}
}
...
struct ContentView: View {
@State var bot: Bot? = nil
@State var progress: CGFloat = 0
func updateProgress(_ progress: Double) {
self.progress = CGFloat(progress)
}
var body: some View {
if let bot {
BotView(bot)
} else {
ProgressView(value: progress) {
Text("loading huggingface model...")
} currentValueLabel: {
Text(String(format: "%.2f%%", progress * 100))
}
.padding()
.onAppear() { Task {
let bot = await Bot(updateProgress)
await MainActor.run { self.bot = bot }
} }
}
}
}
```
## Structured Output with @Generatable
The `@Generatable` macro enables **100% reliable** type-safe structured output generation. No more struggling with prompting to get output in the format you want—it works every time and allows true programmatic flow. Simply annotate your Swift structs and enums to automatically generate JSON schemas that guide the model to produce valid, structured responses:
```swift
@Generatable
struct Person {
let name: String
let age: Int
let occupation: String
let personality: String
}
let bot = LLM(from: Bundle.main.url(forResource: "model", withExtension: "gguf")!, template: .chatML("You are helpful."))
let result = try await bot.respond(to: "Create a fictional character", as: Person.self)
let person = result.value // Guaranteed to be a valid Person struct
print(person.name) // "Alice"
print(person.age) // 28
```
The macro works with structs, enums, arrays, and supports nested Generatable structures:
```swift
@Generatable
enum Priority {
case low, medium, high, urgent
}
@Generatable
struct Address {
let street: String
let city: String
let zipCode: String
}
@Generatable
struct Task {
let title: String
let priority: Priority
let assignee: Person // Nested Generatable struct
}
@Generatable
struct Project {
let name: String
let tasks: [Task] // Arrays of Generatable structs
let teamLead: Person // Nested Generatable types
let office: Address // Multiple levels of nesting
}
let result = try await bot.respond(to: "Create a software project plan", as: Project.self)
```
The macro automatically:
- Generates JSON schema for structs and enums
- Adds Codable conformance and CaseIterable for enums
- Handles nested Generatable structures and arrays
- Provides automatic validation
- Returns both the parsed object and raw JSON output
> [!TIP]
> Check `LLMTests.swift` for more comprehensive examples and use cases of `@Generatable`.
## Function Calling (Tools)
`LLM.swift` supports function calling (tool use), letting the model call into your own Swift code and use the result to answer the user. A tool is a type conforming to the `Tool` protocol, with its arguments declared as a nested type using the same `@Generatable` macro used for structured output—their JSON schema is derived automatically and advertised to the model:
```swift
struct GetWeather: Tool {
let description = "Get the current weather for a city"
@Generatable
struct Arguments {
let city: String
}
func call(_ arguments: Arguments) async throws -> String {
"It is sunny and 22 degrees celsius in \(arguments.city)."
}
}
let bot = try await LLM(from: HuggingFaceModel("unsloth/Qwen3-0.6B-GGUF", .Q4_K_M))!
bot.systemPrompt = "You are a helpful assistant."
bot.tools = [GetWeather()]
await bot.respond(to: "What is the weather in Seoul right now?")
```
The tool's `name` defaults to the type name; provide a `name` property to override it.
Tool calls use the model's native tool-call format, rendered and parsed by `llama.cpp`'s `common/chat` machinery and driven by the gguf's own template, so any model family with tool support in its chat template works, with no per-model configuration. This requires using the embedded-template path, so don't set a manual `template` when you want to use tools.
Every tool call the model makes is recorded in `@Published var toolCalls: [ToolCall]`, including its name, arguments JSON, and result, so you can display it in your UI. Multiple tool-call rounds are supported, up to `maxToolTurns` (default `4`). If a tool throws, or its arguments fail to decode, the error text is fed back to the model so it can recover—`LLM.swift` itself does not throw in that case.
> [!TIP]
> Check `LLMTests.swift` for more comprehensive examples and use cases of tool calling.
## Usage
Add the package using SPM:
```swift
dependencies: [
.package(url: "https://github.com/eastriverlee/LLM.swift/", branch: "main"),
],
```
## Overview
`LLM.swift` started as a lightweight abstraction layer over [`llama.cpp`](https://github.com/ggerganov/llama.cpp), and has evolved into a comprehensive Swift library with advanced features like `@Generatable` that allows users to control LLMs programmatically. It stays as performant as possible while always being up to date—any model that works on [`llama.cpp`](https://github.com/ggerganov/llama.cpp) should work with this library as well.
The core implementation is in `LLM.swift`, making it easy to understand and extend the library for your specific needs.
there are some lines that are especially worth paying your attention to to grasp its internal structure:
```swift
public typealias Chat = (role: Role, content: String)
public enum Role {
case user
case bot
}
```
```swift
public var history: [Chat]
public var preprocess: (_ input: String, _ history: [Chat]) -> String = { input, _ in return input }
public var postprocess: (_ output: String) -> Void = { print($0) }
public var update: (_ outputDelta: String?) -> Void = { _ in }
public func respond(to input: String, with makeOutputFrom: @escaping (AsyncStream) async -> String) async {
guard isAvailable else { return }
isAvailable = false
self.input = input
let processedInput = preprocess(input, history)
let response = getResponse(from: processedInput)
let output = await makeOutputFrom(response)
history += [(.user, input), (.bot, output)]
if historyLimit < history.count {
history.removeFirst(2)
}
postprocess(output)
isAvailable = true
}
open func respond(to input: String) async {
await respond(to: input) { [self] response in
await setOutput(to: "")
for await responseDelta in response {
update(responseDelta)
await setOutput(to: output + responseDelta)
}
update(nil)
let trimmedOutput = output.trimmingCharacters(in: .whitespacesAndNewlines)
await setOutput(to: trimmedOutput.isEmpty ? "..." : trimmedOutput)
return output
}
}
```
> [!TIP]
> as you can see, `func respond(to input: String) async` has an `open` access, so that you can override it when your class inherits `LLM` and want some fine grained control over.
there are three functions users can define when initializing `LLM` class:
* `var preprocess: (_ input: String, _ history: [Chat], _ thinking: ThinkingMode) -> String`
* `var postprocess: (_ output: String) -> Void`
* `var update: (_ outputDelta: String?) -> Void`
* `var updateThinking: (_ thinkingDelta: String?) -> Void`
they are used in `respond` function.
### preprocess
`preprocess` is commonly used for making the user input conform to a chat template. if you don't provide this, `LLM` will just work as a completion model.
for example, this is the `ChatML` template, that is adopted by many chat models:
```
<|im_start|>system
SYSTEM PROMPT<|im_end|>
<|im_start|>user
USER PROMPT<|im_end|>
<|im_start|>assistant
```
to use this chat format, you should use a function that goes like this:
```swift
self.preprocess = { input, history, thinking in
let systemPrompt = "You are a sentient AI with emotions."
var processed = "<|im_start|>system\n\(systemPrompt)<|im_end|>\n"
for chat in history {
processed += "<|im_start|>\(chat.role == .user ? "user" : "assistant")\n\(chat.content)<|im_end|>\n"
}
processed += "<|im_start|>user\n\(input)<|im_end|>\n"
processed += "<|im_start|>assistant\n"
// Handle thinking mode
if thinking == .enabled {
processed += ""
} else if thinking == .none {
processed += ""
}
return processed
}
```
but for convenience, you can use `Template` struct that is specifically made for this:
```swift
// you can use the static function that is already available for this:
self.preprocess = Template.chatML("You are a sentient AI with emotions.").preprocess
// or even better
// you can set [template] property right away, so that it handles [preprocess] and [stopSequence] both:
self.template = .chatML("You are a sentient AI with emotions.")
// which is the same thing as:
self.template = Template(
system: ("<|im_start|>system\n", "<|im_end|>\n"),
user: ("<|im_start|>user\n", "<|im_end|>\n"),
bot: ("<|im_start|>assistant\n", "<|im_end|>\n"),
stopSequence: "<|im_end|>",
systemPrompt: "You are a sentient AI with emotions."
)
```
> [!TIP]
> checking `LLMTests.swift` will help you understand how `preprocess` works better.
### postprocess
`postprocess` can be used for executing according to the `output` just made using user input.
the default is set to `{ print($0) }`, so that it will print the output when it's finished generating by meeting `EOS` or `stopSequence`.
this has many usages. for instance, this can be used to implement your own function calling logic.
### update
if you use regular `func respond(to input: String) async` `update` function that you set will get called every time when you get `outputDelta`.
`outputDelta` is `nil` when it stops generating the output.
if you want more control over everything you can use `func respond(to input: String, with makeOutputFrom: @escaping (AsyncStream) async -> String) async` instead, which the aforementioned function uses internally, to define your own version of `makeOutputFrom` function that is used to make `String` typed output out of `AsyncStream` and add to its history. in this case, `update` function will be ignored unless you use it. check `func respond(to input: String) async` implementation shown above to understand how it works.
## Thinking Process (Chain of Thought)
You can access the model's thinking process separately from the final response. This is useful for models which support Chain of Thought.
```swift
// Enable thinking when calling respond (defaults to .none)
// If set to .none, it uses the "nothink" technique to suppress thinking tokens
await bot.respond(to: input, thinking: .enabled)
// Access the accumulated thinking content
print(bot.thinking)
```
The `thinking` property on your bot instance will contain the accumulated thought process, while `output` contains the final response. The library automatically handles the `` tags and separation.
### ThinkingMode
- `.enabled`: Forces the model to think by appending the thinking start token. Parses and separates thinking content from the response.
- `.suppressed`: Forces the model to skip thinking by appending empty thinking tokens (nothink technique).
- `.none`: (Default) No special handling for thinking tokens.
## Embeddings
LLM.swift supports text embeddings for semantic similarity and search applications:
```swift
// Generate embeddings for text
let embeddings1 = try await bot.getEmbeddings("Hello world")
let embeddings2 = try await bot.getEmbeddings("Hi there")
let embeddings3 = try await bot.getEmbeddings("Goodbye")
// Compare similarity (returns 0.0 to 1.0)
let similarity = embeddings1.compare(with: embeddings2)
print(similarity) // 0.8 (high similarity)
// Find most similar embedding
let mostSimilar = embeddings1.findMostSimilar(in: embeddings2, embeddings3)
print(mostSimilar == embeddings2) // true
```
The `Embeddings` struct provides:
- `compare(with:)` - Computes cosine similarity between two embeddings (0.0 to 1.0)
- `findMostSimilar(in:)` - Returns the most similar embedding from a set of candidates
- `Equatable` conformance for direct comparison