Too many documents to read?
…or drag into ChatGPT?
FastCorpus provides specialized, agent-ready tools for text analysis. Small, fast and sustainable, so you don’t need a general-purpose LLM to make sense of your documents.
import fastcorpus as fc
corp = fc.Corpus.from_parquet(
"documents.parquet", text_field="text"
)
dfmat = (
corp.tokens(remove_punct=True)
.stopwords("en")
.group(by="party")
.dfm()
)
dfmat.topfeatures(n=10).collect()
80–90% of data is unstructured text.
The usual answer is to send it all to a large language model. That works, but it is the container ship for a parcel delivery.
General-purpose LLMs
- Big models
- Many tokens
- Do everything
- Not very efficient
FastCorpus
- Small models
- Few tokens
- Does text analysis
- Sustainable
Faster, lighter, lower energy.
Corpus → tokens → document-feature matrix on 2.5 million parliamentary documents, 24 threads.
| Corpus | Tool | Time | vs FastCorpus |
|---|---|---|---|
| 2.5 million documents | FastCorpus | 23.7 s | |
| 2.5 million documents | NLTK + SciPy | 96.6 s | 4.1× slower |
| 2.5 million documents | spaCy + textacy | 105 s | 4.4× slower |
| 2.5 million documents | quanteda | 808 s | 34.1× slower |
Two ways in.
FastCorpus is open source (Apache-2.0). It is not on PyPI yet, so build it from source with a Rust toolchain (1.95+).
Python
A quanteda-style API (Corpus → Tokens → DFM), lazy by default, with no runtime dependencies. Requires Python 3.12+ and uv.
git clone https://github.com/seankellyhp/fastcorpus
cd fastcorpus/fastcorpus-py
uv venv --python 3.12
uv pip install maturin
uv run maturin develop --release
AI agents (MCP)
An MCP server that lets an agent search and compare a corpus without reading the raw text into its context.
cargo build --release -p fastcorpus-mcp
# MCP client config
{ "mcpServers": { "fastcorpus": {
"command": "/path/to/target/release/fastcorpus-mcp"
} } }
Read the corpus without reading the corpus.
Point FastCorpus at your documents, or hand it to your agent, and get to the patterns in seconds instead of millions of tokens.