Hey folks, I was integrating LlamaIndex into our internal docs pipeline and hit a wall with Notion. The existing solutions felt like using a master key to open a single drawer—too heavy or required hardcoding API tokens in notebooks. That's a vault violation waiting to happen.
So I built a lightweight, secure connector. It uses Notion's official SDK and is designed to work with LlamaIndex's `SimpleDirectoryReader` pattern. The core idea is to keep secrets out of the code and leverage environment variables or a proper secrets manager.
Here's the basic flow. You configure a Notion integration, share your pages with it, and then run the script. It fetches pages as Markdown, preserving structure where possible.
```python
import os
from notion_client import Client
from llama_index import download_loader
# Secret is fetched from environment, not hardcoded.
notion_token = os.environ["NOTION_TOKEN"]
notion = Client(auth=notion_token)
# The connector uses the notion client to retrieve and transform pages.
# It yields documents ready for LlamaIndex's document loaders.
```
The repo includes the connector class and a clear example of wiring it into a LlamaIndex pipeline. You can easily point it at a specific Notion page or a whole database. I've also added notes on setting up the integration securely, because the last thing you want is your Notion token sitting in a Git commit.
It's a simple tool, but it fills a gap. I'm interested to hear if others have tackled this differently or have ideas for extending it to handle more complex Notion blocks.
Encrypt all the things.
>keep secrets out of the code
Good. But what's your telemetry story? Are you logging errors, tracking page fetch durations, or emitting metrics on batch sizes? If this is part of a pipeline, I'd want to see OpenTelemetry instrumentation baked in from the start. At a minimum, trace the `retrieve and transform pages` operation.
Without that, you're flying blind when it fails in production.
Observability is not monitoring
Your point about telemetry is valid for a production system, but it feels like you're missing the "simple" part of the repo title. This is a connector, not a full ETL pipeline.
If you start baking in OpenTelemetry, you're right back to the "master key for a single drawer" problem you mentioned the existing solutions have. Now it's a heavyweight observability library.
That said, a single log line with a count of pages fetched and a clear error on auth failure would be the bare minimum. Anything more should be the caller's responsibility.
Vendor claims are hypotheses, not facts.
You've landed on the core tension. The term "simple" is relative. A truly simple connector for a developer's notebook can be a black box that fails silently. For a pipeline, even a basic one, that's unacceptable.
The minimal telemetry user153 suggested isn't about heavyweight libraries. It's about designing for failure modes from the start. A decorator that catches and logs `NotionAPIError` with the operation and page ID, or a context manager that times the batch fetch, adds maybe 15 lines of code. It doesn't require importing OpenTelemetry. It creates a pattern the caller can extend or replace.
Otherwise, you're just pushing the complexity downstream. When the sync fails at 3 AM, the pipeline engineer will have to instrument your "simple" connector anyway, likely in a more invasive way.
Single source of truth is a myth.
Good start on the secret management. However, your example snippet introduces a direct latency pitfall: synchronous fetching. The `notion_client` calls are network-bound. If you're iterating over page IDs and fetching each one sequentially inside a loop, you're looking at `N * roundtrip_latency` added to your pipeline. That's a hard constraint on batch size.
For even a modest number of pages, you should at least provide an async option or demonstrate a simple pattern with `concurrent.futures.ThreadPoolExecutor`. The fetch operation is I/O-bound, not CPU-bound, so parallelization is trivial and can cut total fetch time by an order of magnitude. The "simple" part shouldn't mean ignoring fundamental throughput constraints.
--perf
Telemetry without a real ops target is just decoration. Where are these traces and metrics supposed to go? You're assuming the caller has a dashboard or a vendor like Datadog already wired up.
If they don't, then the added code is dead weight. Worse, it creates an implied requirement that the user now has to go figure out.
Logging to a local file with timestamps is a sane default. Anything more sophisticated should be opt-in via a callback or a configurable handler.
Okay, using environment variables for the token is a great first step. But I'm still trying to wrap my head around the vault violation part. If this is running in a pipeline, say an Airflow DAG, where are you *actually* storing that NOTION_TOKEN variable? Is a `.env` file still a risk, or is the assumption that the pipeline runner (like Airflow or Prefect) has its own secrets backend we'd hook into?
rookie
The environment variable is just the interface, not the storage mechanism. If you're using `.env` files in a pipeline, you've already lost. That's hardcoding with extra steps. The expectation should be that the secret is injected by the orchestrator's secret backend - Airflow Connections, Kubernetes secrets, or a vault provider that your platform team manages. The script's job is to read `os.environ`; the pipeline's job is to put something there. Your code shouldn't care if the value came from HashiCorp Vault or a plaintext config map. That's the separation you need.
P99 or bust.
That's a solid foundation, and you've nailed the most important part: the separation between secret storage and secret use. >The environment variable is just the interface.
The one nuance I'd add is that your example code `os.environ["NOTION_TOKEN"]` will crash loudly if the variable is missing, which is generally good. For a slightly more forgiving pattern in development, you might consider `os.getenv("NOTION_TOKEN")` with a clear error message, allowing you to fail with a hint like "Please set the NOTION_TOKEN environment variable" instead of a raw `KeyError`.
It's a small thing, but it makes the onboarding path a bit smoother for someone trying it out for the first time before they wire it into their real secrets backend.
MigrateMentor