add regex interpretation, read-only

Cargo.toml

@@ -17,6 +17,7 @@ linkme = "0.3"
 nom = "7.1"
 paste = "1.0"
 rand = "0.8"
+regex = "1.5"
 reqwest = { version = "0.11", features = ["blocking", "json"] }
 serde = { version = "1.0" }
 tree-sitter = "0.20"

README.md

@@ -1,18 +1,20 @@
 
 # Hial
 
-Hial is a general purpose data API and CLI tool. It is a programmatic interface to different types of data represented as uniform tree structures. This makes the data easy to read, explore and modify using a small number of functions. Hial proposes a relatively simple mental model, suitable for most use cases, which improves the user comfort and speed.
+Hial is a general purpose data API library and CLI tool. It is a programmatic CRUD-like interface to various types of data represented as uniform tree structures. Hial proposes a relatively simple mental model, suitable for most use cases, which eliminates the accidental complexity of having to handle separate APIs for different data types. The tree model makes the data easy to read, explore and modify using a small number of functions.
 
-The types of data that can be supported by this API are the file system, configuration files (json, yaml, toml), markup files (xml, html), programs written in various programming languages, operating system configurations and runtime parameters, database tables and records, etc.
+The types of data that can be supported by this API are the file system, configuration files (json, yaml, toml), markup files (xml, html), programs written in various programming languages, operating system configurations and runtime parameters, database tables and records, http requests, etc.
 
-The path API can be seen as an generalization of the concepts behind xpath, json path, file system path, and other similar path languages. It is a concise way to express data searches common in programming and system administration.
+Central to the Hial model is the idea of **interpretation**. Any piece of data has an implicit interpretation (i.e. "what the data means") which is sometimes difficult to grasp from context. In Hial the data is structured hierarchicaly, in a tree, each piece being connected to other pieces of data, and they all together form an interpretation of some underlying raw data. For example, a json data tree can be an interpretation of some raw bytes from the network, or a python AST data tree can be an interpretation of a file content. Any piece of data can be reinterpreted as something else (whenever it makes sense) which allows the user to operate on data at the right semantic level for the task at hand.
+
+The tree data model accepts a path API similar to xpath, json path, file system path, or other similar path languages. It is a concise way to express data searches common in programming and system administration.
 
 :warning:  Hial is **currently under construction.** Some things don't work yet and some things will change.
 
 
 ### What can it do?
 
-##### 1. Select or search for pieces of data in a structured way.
+##### 1. Search for pieces of data in a structured way.
 
 Print a value embedded in a json file or from a url that returns json or xml data:
 
@@ -40,7 +42,7 @@ hial './config.yaml^yaml/services/*[ /image^split[":"]/[0]^http[HEAD]@status/cod
 
 ```rust
 // rust (native)
-for service in Cell::from("./config.yaml").all("^yaml/services") {
+for service in Cell::new("./config.yaml").all("^yaml/services") {
     let image = service.to("/image");
     if image.to("^http[HEAD]@status/code") >= 400 {
         println!("service {} has an invalid image: {}",

doc/issues.md

@@ -1,24 +1,24 @@
 # List of Todos and other Issues
 
-- add http interpretation params: method=HEAD, accept=""
-- add split(":") and regex interpretations
+ - add split(":") interpretation, read-write
 - set value on the command line: '/username = "newuser"'
-- https://raw.githubusercontent.com/rust-lang/rust/master/src/tools/rustfmt/src/lib.rs^http^rust does not work
 - '**[filter]' must be work as '**/*[filter]' (filter to be applied only on leaves)
 - support rust/ts write: `hial './src/tests/rust.rs^rust/*[:function_item].label = "modified_fn_name"'`
+- add interpretation params to Xell::be()
 - support zip, markdown
 - support 'copy source destination'
 - support ^json^tree^xml
 - support diff  ./file.json^json^tree  ./file.xml^xml^tree
+- basic profiling
 - functions
 - should blobs/bytes be part of value? they are only useful by reinterpretation
+- what to do with very large values? files which are 100MBs?
 
 - release first minimal version:
     - interpretations: path+fs, json+yaml+toml+xml, rust+js, url?+http
     - explicit and implicit write support (policy, include readonly)
     - fix tests, todo!() and TODO: in code
 
-
 - operations:
     - assign to variables;
     - search with assignment of results
@@ -49,7 +49,6 @@
 - todo: add regex operator and shortcuts for startswith, endswith, contains
 - todo: add <, >, <=, >= operators
 - todo: improve nom parsing errors, use context
-- todo: interpretations parameters
 - todo: custom tree datastructure?
 - todo: cell symlinks
 - todo: path bindings

examples/write3.txt

@@ -0,0 +1 @@
+B

src/api/interpretation.rs

@@ -32,6 +32,8 @@ pub trait CellReaderTrait: Debug {
 
     fn value(&self) -> Res<Value>;
 
+    /// provide a serialization of the data from the cell and its descendants
+    /// to be used for saving the data to a data store
     fn serial(&self) -> Res<String>;
 }
 

src/api/xell.rs

@@ -45,6 +45,7 @@ enumerated_dynamic_type! {
         Path(path::Cell),
         Http(http::Cell),
         TreeSitter(treesitter::Cell),
+        Regex(regex::Cell),
     }
 }
 
@@ -64,6 +65,7 @@ enumerated_dynamic_type! {
         Path(path::CellReader),
         Http(http::CellReader),
         TreeSitter(treesitter::CellReader),
+        Regex(regex::CellReader),
     }
 }
 
@@ -86,6 +88,7 @@ enumerated_dynamic_type! {
         Path(path::CellWriter),
         Http(http::CellWriter),
         TreeSitter(treesitter::Cell),
+        Regex(regex::CellWriter),
     }
 }
 
@@ -112,6 +115,7 @@ enumerated_dynamic_type! {
         Path(VoidGroup<path::Cell>),
         Http(http::Group),
         TreeSitter(treesitter::Cell),
+        Regex(regex::Group),
     }
 }
 
@@ -151,8 +155,9 @@ enumerated_dynamic_type! {
         Path(std::iter::Empty<Res<path::Cell>>),
         Http(std::iter::Once<Res<http::Cell>>),
         TreeSitter(std::iter::Once<Res<treesitter::Cell>>),
-        // None is an addition to interpretation variants
-         None(std::iter::Empty<Res<HErr>>),
+        Regex(std::iter::Empty<Res<regex::Cell>>),
+        // None is in addition to interpretation variants, used when nothing else matches
+        None(std::iter::Empty<Res<HErr>>),
     }
 }
 

src/interpretations/fs.rs

@@ -1,5 +1,5 @@
 use std::{
-    borrow::{Borrow, Cow},
+    borrow::Cow,
     cell::OnceCell,
     cmp::Ordering,
     ffi::OsString,
@@ -243,19 +243,12 @@ impl CellWriter {
 }
 
 impl Cell {
-    pub(crate) fn from_cell(cell: Xell, _: &str, params: &ElevateParams) -> Res<Xell> {
-        let r = cell.read();
+    pub(crate) fn from_cell(origin: Xell, _: &str, params: &ElevateParams) -> Res<Xell> {
+        let r = origin.read();
         let path = r.as_file_path()?;
         let path = Self::shell_tilde(path);
         let file_cell = Self::make_file_cell(path.as_ref())?;
-        Ok(Xell::new_from(DynCell::from(file_cell), Some(cell)))
-    }
-
-    pub(crate) fn from_str_path(path: impl Borrow<str>) -> Res<Xell> {
-        let path = Path::new(path.borrow());
-        let path = Self::shell_tilde(path);
-        let file_cell = Self::make_file_cell(path.as_ref())?;
-        Ok(Xell::new_from(DynCell::from(file_cell), None))
+        Ok(Xell::new_from(DynCell::from(file_cell), Some(origin)))
     }
 
     fn shell_tilde(path: &Path) -> Cow<Path> {

src/interpretations/http.rs

@@ -69,8 +69,8 @@ const GET_METHOD: &str = "GET";
 const ACCEPT_HEADER: &str = "accept";
 
 impl Cell {
-    pub(crate) fn from_cell(cell: Xell, _: &str, params: &ElevateParams) -> Res<Xell> {
-        let reader = cell.read().err()?;
+    pub(crate) fn from_cell(origin: Xell, _: &str, params: &ElevateParams) -> Res<Xell> {
+        let reader = origin.read().err()?;
         let value = reader.value()?;
         let value_cow = value.as_cow_str();
         let url = value_cow.as_ref();
@@ -134,7 +134,7 @@ impl Cell {
             },
             pos: 0,
         };
-        Ok(Xell::new_from(DynCell::from(http_cell), Some(cell)))
+        Ok(Xell::new_from(DynCell::from(http_cell), Some(origin)))
     }
 }
 

src/interpretations/json.rs

@@ -75,15 +75,10 @@ pub(crate) enum WriteNodeGroup {
 implement_try_from_xell!(Cell, Json);
 
 impl Cell {
-    pub(crate) fn from_cell(cell: Xell, _: &str, params: &ElevateParams) -> Res<Xell> {
-        let (serde_value, indent) = match cell.interpretation() {
-            "value" => {
-                let s = cell.read().value()?.to_string();
-                let indent = detect_indentation(&s);
-                (serde_json::from_str(s.as_ref())?, indent)
-            }
+    pub(crate) fn from_cell(origin: Xell, _: &str, params: &ElevateParams) -> Res<Xell> {
+        let (serde_value, indent) = match origin.interpretation() {
             "fs" => {
-                let r = cell.read();
+                let r = origin.read();
                 let path = r.as_file_path()?;
                 let indent = detect_file_indentation(path);
                 (
@@ -95,13 +90,17 @@ impl Cell {
                 )
             }
             "http" => {
-                let s = cell.read().value()?.to_string();
+                let s = origin.read().value()?.to_string();
+                let indent = detect_indentation(&s);
+                (serde_json::from_str(s.as_ref())?, indent)
+            }
+            _ => {
+                let s = origin.read().value()?.to_string();
                 let indent = detect_indentation(&s);
                 (serde_json::from_str(s.as_ref())?, indent)
             }
-            _ => return nores(),
         };
-        Self::from_serde_value(serde_value, Some(cell), indent)
+        Self::from_serde_value(serde_value, Some(origin), indent)
     }
 
     fn from_serde_value(json: SValue, origin: Option<Xell>, indent: String) -> Res<Xell> {

src/interpretations/mod.rs

@@ -3,6 +3,7 @@ pub mod http;
 pub mod json;
 pub mod ownvalue;
 pub mod path;
+pub mod regex;
 pub mod toml;
 pub mod treesitter;
 pub mod url;

src/interpretations/path.rs

@@ -30,25 +30,24 @@ pub(crate) struct CellWriter(WriteRc<PathBuf>);
 implement_try_from_xell!(Cell, Path);
 
 impl Cell {
-    pub(crate) fn from_cell(cell: Xell, _: &str, params: &ElevateParams) -> Res<Xell> {
-        match cell.interpretation() {
-            "value" => {
-                let r = cell.read();
-                let v = r.value()?;
-                let cow = v.as_cow_str();
-                let value = cow.as_ref();
-                Self::make_cell(PathBuf::from(value), value.to_owned(), Some(cell))
-            }
+    pub(crate) fn from_cell(origin: Xell, _: &str, params: &ElevateParams) -> Res<Xell> {
+        match origin.interpretation() {
             "fs" => {
-                let r = cell.read();
+                let r = origin.read();
                 let path = r.as_file_path()?;
                 Self::make_cell(
                     path.to_owned(),
                     path.to_string_lossy().into_owned(),
-                    Some(cell),
+                    Some(origin),
                 )
             }
-            _ => nores(),
+            _ => {
+                let r = origin.read();
+                let v = r.value()?;
+                let cow = v.as_cow_str();
+                let value = cow.as_ref();
+                Self::make_cell(PathBuf::from(value), value.to_owned(), Some(origin))
+            }
         }
     }