feat: record messages from user in ~/.codex/history.jsonl (#939)

This is a large change to support a "history" feature like you would
expect in a shell like Bash.

History events are recorded in `$CODEX_HOME/history.jsonl`. Because it
is a JSONL file, it is straightforward to append new entries (as opposed
to the TypeScript file that uses `$CODEX_HOME/history.json`, so to be
valid JSON, each new entry entails rewriting the entire file). Because
it is possible for there to be multiple instances of Codex CLI writing
to `history.jsonl` at once, we use advisory file locking when working
with `history.jsonl` in `codex-rs/core/src/message_history.rs`.

Because we believe history is a sufficiently useful feature, we enable
it by default. Though to provide some safety, we set the file
permissions of `history.jsonl` to be `o600` so that other users on the
system cannot read the user's history. We do not yet support a default
list of `SENSITIVE_PATTERNS` as the TypeScript CLI does:


https://github.com/openai/codex/blob/3fdf9df1335ac9501e3fb0e61715359145711e8b/codex-cli/src/utils/storage/command-history.ts#L10-L17

We are going to take a more conservative approach to this list in the
Rust CLI. For example, while `/\b[A-Za-z0-9-_]{20,}\b/` might exclude
sensitive information like API tokens, it would also exclude valuable
information such as references to Git commits.

As noted in the updated documentation, users can opt-out of history by
adding the following to `config.toml`:

```toml
[history]
persistence = "none" 
```

Because `history.jsonl` could, in theory, be quite large, we take a[n
arguably overly pedantic] approach in reading history entries into
memory. Specifically, we start by telling the client the current number
of entries in the history file (`history_entry_count`) as well as the
inode (`history_log_id`) of `history.jsonl` (see the new fields on
`SessionConfiguredEvent`).

The client is responsible for keeping new entries in memory to create a
"local history," but if the user hits up enough times to go "past" the
end of local history, then the client should use the new
`GetHistoryEntryRequest` in the protocol to fetch older entries.
Specifically, it should pass the `history_log_id` it was given
originally and work backwards from `history_entry_count`. (It should
really fetch history in batches rather than one-at-a-time, but that is
something we can improve upon in subsequent PRs.)

The motivation behind this crazy scheme is that it is designed to defend
against:

* The `history.jsonl` being truncated during the session such that the
index into the history is no longer consistent with what had been read
up to that point. We do not yet have logic to enforce a `max_bytes` for
`history.jsonl`, but once we do, we will aspire to implement it in a way
that should result in a new inode for the file on most systems.
* New items from concurrent Codex CLI sessions amending to the history.
Because, in absence of truncation, `history.jsonl` is an append-only
log, so long as the client reads backwards from `history_entry_count`,
it should always get a consistent view of history. (That said, it will
not be able to read _new_ commands from concurrent sessions, but perhaps
we will introduce a `/` command to reload latest history or something
down the road.)

Admittedly, my testing of this feature thus far has been fairly light. I
expect we will find bugs and introduce enhancements/fixes going forward.

Author: bolinfest

codex-rs/Cargo.lock

@@ -23,7 +23,9 @@ This folder is the root of a Cargo workspace. It contains quite a bit of experim
 
 ## Config
 
-The CLI can be configured via `~/.codex/config.toml`. It supports the following options:
+The CLI can be configured via a file named `config.toml`. By default, configuration is read from `~/.codex/config.toml`, though the `CODEX_HOME` environment variable can be used to specify a directory other than `~/.codex`.
+
+The `config.toml` file supports the following options:
 
 ### model
 
@@ -297,6 +299,17 @@ To have Codex use this script for notifications, you would configure it via `not
 notify = ["python3", "/Users/mbolin/.codex/notify.py"]
 ```
 
+### history
+
+By default, Codex CLI records messages sent to the model in `$CODEX_HOME/history.jsonl`. Note that on UNIX, the file permissions are set to `o600`, so it should only be readable and writable by the owner.
+
+To disable this behavior, configure `[history]` as follows:
+
+```toml
+[history]
+persistence = "none"  # "save-all" is the default value
+```
+
 ### project_doc_max_bytes
 
 Maximum number of bytes to read from an `AGENTS.md` file to include in the instructions sent with the first turn of a session. Defaults to 32 KiB.

codex-rs/README.md

@@ -20,6 +20,7 @@ codex-mcp-client = { path = "../mcp-client" }
 dirs = "6"
 env-flags = "0.1.1"
 eventsource-stream = "0.2.3"
+fs2 = "0.4.3"
 fs-err = "3.1.0"
 futures = "0.3"
 mcp-types = { path = "../mcp-types" }

codex-rs/core/Cargo.toml

@@ -110,6 +110,7 @@ impl Codex {
             cwd: config.cwd.clone(),
         };
 
+        let config = Arc::new(config);
         tokio::spawn(submission_loop(config, rx_sub, tx_event, ctrl_c));
         let codex = Codex {
             next_id: AtomicU64::new(0),
@@ -483,11 +484,14 @@ impl AgentTask {
 }
 
 async fn submission_loop(
-    config: Config,
+    config: Arc<Config>,
     rx_sub: Receiver<Submission>,
     tx_event: Sender<Event>,
     ctrl_c: Arc<Notify>,
 ) {
+    // Generate a unique ID for the lifetime of this Codex session.
+    let session_id = Uuid::new_v4();
+
     let mut sess: Option<Arc<Session>> = None;
     // shorthand - send an event when there is no active session
     let send_no_session_event = |sub_id: String| async {
@@ -608,7 +612,9 @@ async fn submission_loop(
 
                 // Attempt to create a RolloutRecorder *before* moving the
                 // `instructions` value into the Session struct.
-                let session_id = Uuid::new_v4();
+                // TODO: if ConfigureSession is sent twice, we will create an
+                // overlapping rollout file. Consider passing RolloutRecorder
+                // from above.
                 let rollout_recorder =
                     match RolloutRecorder::new(&config, session_id, instructions.clone()).await {
                         Ok(r) => Some(r),
@@ -633,10 +639,19 @@ async fn submission_loop(
                     rollout: Mutex::new(rollout_recorder),
                 }));
 
+                // Gather history metadata for SessionConfiguredEvent.
+                let (history_log_id, history_entry_count) =
+                    crate::message_history::history_metadata(&config).await;
+
                 // ack
                 let events = std::iter::once(Event {
                     id: sub.id.clone(),
-                    msg: EventMsg::SessionConfigured(SessionConfiguredEvent { session_id, model }),
+                    msg: EventMsg::SessionConfigured(SessionConfiguredEvent {
+                        session_id,
+                        model,
+                        history_log_id,
+                        history_entry_count,
+                    }),
                 })
                 .chain(mcp_connection_errors.into_iter());
                 for event in events {
@@ -691,6 +706,46 @@ async fn submission_loop(
                     other => sess.notify_approval(&id, other),
                 }
             }
+            Op::AddToHistory { text } => {
+                let id = session_id;
+                let config = config.clone();
+                tokio::spawn(async move {
+                    if let Err(e) = crate::message_history::append_entry(&text, &id, &config).await
+                    {
+                        tracing::warn!("failed to append to message history: {e}");
+                    }
+                });
+            }
+
+            Op::GetHistoryEntryRequest { offset, log_id } => {
+                let config = config.clone();
+                let tx_event = tx_event.clone();
+                let sub_id = sub.id.clone();
+
+                tokio::spawn(async move {
+                    // Run lookup in blocking thread because it does file IO + locking.
+                    let entry_opt = tokio::task::spawn_blocking(move || {
+                        crate::message_history::lookup(log_id, offset, &config)
+                    })
+                    .await
+                    .unwrap_or(None);
+
+                    let event = Event {
+                        id: sub_id,
+                        msg: EventMsg::GetHistoryEntryResponse(
+                            crate::protocol::GetHistoryEntryResponseEvent {
+                                offset,
+                                log_id,
+                                entry: entry_opt,
+                            },
+                        ),
+                    };
+
+                    if let Err(e) = tx_event.send(event).await {
+                        tracing::warn!("failed to send GetHistoryEntryResponse event: {e}");
+                    }
+                });
+            }
         }
     }
     debug!("Agent loop exited");

codex-rs/core/src/codex.rs

@@ -81,6 +81,30 @@ pub struct Config {
     /// Directory containing all Codex state (defaults to `~/.codex` but can be
     /// overridden by the `CODEX_HOME` environment variable).
     pub codex_home: PathBuf,
+
+    /// Settings that govern if and what will be written to `~/.codex/history.jsonl`.
+    pub history: History,
+}
+
+/// Settings that govern if and what will be written to `~/.codex/history.jsonl`.
+#[derive(Deserialize, Debug, Clone, PartialEq, Default)]
+pub struct History {
+    /// If true, history entries will not be written to disk.
+    pub persistence: HistoryPersistence,
+
+    /// If set, the maximum size of the history file in bytes.
+    /// TODO(mbolin): Not currently honored.
+    pub max_bytes: Option<usize>,
+}
+
+#[derive(Deserialize, Debug, Clone, PartialEq, Default)]
+#[serde(rename_all = "kebab-case")]
+pub enum HistoryPersistence {
+    /// Save all history entries to disk.
+    #[default]
+    SaveAll,
+    /// Do not write history to disk.
+    None,
 }
 
 /// Base config deserialized from ~/.codex/config.toml.
@@ -130,6 +154,10 @@ pub struct ConfigToml {
     /// Named profiles to facilitate switching between different configurations.
     #[serde(default)]
     pub profiles: HashMap<String, ConfigProfile>,
+
+    /// Settings that govern if and what will be written to `~/.codex/history.jsonl`.
+    #[serde(default)]
+    pub history: Option<History>,
 }
 
 impl ConfigToml {
@@ -297,6 +325,8 @@ impl Config {
             }
         };
 
+        let history = cfg.history.unwrap_or_default();
+
         let config = Self {
             model: model
                 .or(config_profile.model)
@@ -320,6 +350,7 @@ impl Config {
             model_providers,
             project_doc_max_bytes: cfg.project_doc_max_bytes.unwrap_or(PROJECT_DOC_MAX_BYTES),
             codex_home,
+            history,
         };
         Ok(config)
     }
@@ -468,6 +499,40 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_toml_parsing() {
+        let history_with_persistence = r#"
+[history]
+persistence = "save-all"
+"#;
+        let history_with_persistence_cfg: ConfigToml =
+            toml::from_str::<ConfigToml>(history_with_persistence)
+                .expect("TOML deserialization should succeed");
+        assert_eq!(
+            Some(History {
+                persistence: HistoryPersistence::SaveAll,
+                max_bytes: None,
+            }),
+            history_with_persistence_cfg.history
+        );
+
+        let history_no_persistence = r#"
+[history]
+persistence = "none"
+"#;
+
+        let history_no_persistence_cfg: ConfigToml =
+            toml::from_str::<ConfigToml>(history_no_persistence)
+                .expect("TOML deserialization should succeed");
+        assert_eq!(
+            Some(History {
+                persistence: HistoryPersistence::None,
+                max_bytes: None,
+            }),
+            history_no_persistence_cfg.history
+        );
+    }
+
     /// Deserializing a TOML string containing an *invalid* permission should
     /// fail with a helpful error rather than silently defaulting or
     /// succeeding.
@@ -620,6 +685,7 @@ disable_response_storage = true
                 model_providers: fixture.model_provider_map.clone(),
                 project_doc_max_bytes: PROJECT_DOC_MAX_BYTES,
                 codex_home: fixture.codex_home(),
+                history: History::default(),
             },
             o3_profile_config
         );
@@ -654,6 +720,7 @@ disable_response_storage = true
             model_providers: fixture.model_provider_map.clone(),
             project_doc_max_bytes: PROJECT_DOC_MAX_BYTES,
             codex_home: fixture.codex_home(),
+            history: History::default(),
         };
 
         assert_eq!(expected_gpt3_profile_config, gpt3_profile_config);
@@ -703,6 +770,7 @@ disable_response_storage = true
             model_providers: fixture.model_provider_map.clone(),
             project_doc_max_bytes: PROJECT_DOC_MAX_BYTES,
             codex_home: fixture.codex_home(),
+            history: History::default(),
         };
 
         assert_eq!(expected_zdr_profile_config, zdr_profile_config);

codex-rs/core/src/config.rs

@@ -24,6 +24,7 @@ pub mod landlock;
 mod mcp_connection_manager;
 pub mod mcp_server_config;
 mod mcp_tool_call;
+mod message_history;
 mod model_provider_info;
 pub use model_provider_info::ModelProviderInfo;
 pub use model_provider_info::WireApi;