Embed proposal

[myitcv]

We propose a solution for directly loading files of any type as part of CUE evaluation.

Abstract

Users frequently need to load JSON, YAML, or other types of files into their CUE code. As CUE only supports import declarations that reference CUE packages, users currently resort to the CUE tooling layer (cue cmd) to load non-CUE files, which can be overly complex for their needs. The tooling layer was introduced to handle external influences that make a configuration non-hermetic, typically files.

However, files that are part of a CUE module can be considered hermetic. We aim to make it easier to reference these files.

https://github.com/cue-lang/proposal/blob/main/designs/3264-embed.md

Experimental implementation

v0.10.0-alpha.1 includes an experimental implementation of this initial version of the proposal. It is hoped this will allow for rapid refinement of both the proposal and implementation, based on user feedback and real-world experience.

Enable the embed experiment by setting the CUE_EXPERIMENT environment variable:

export CUE_EXPERIMENT=embed

"Embedding data files in a CUE evaluation" shows a working example using the proposed @embed() attribute.

Known issues

See the issue tracker: embed

Feedback and next steps

This section will be updated with feedback and next steps distilled from responses below

• Implementing ** seemingly a good approach to supporting directories: Embed proposal #3264 (reply in thread)
• Something like type=*auto|text seemingly an acceptable UX for handling mixed (recursive) directory contents with a fallback: Embed proposal #3264 (reply in thread)
• Clarify the position with respect to multiple @embed attributes appearing on a field: cue/interpreter/embed: support embed attribute appearing multiple times for single field declaration? #3294
• Decide on whether we will enable options for path manipulation via the @embed() attribute, or instead "defer" to the path package: Embed proposal #3264 (comment)
• Decide on naming of @embed (vs @read, etc): Embed proposal #3264 (comment)
• Decide on how @embed should behave with respect to symlinks: cue/interpreter/embed: embed should not follow symbolic links #3299 and all: to what extent should CUE support symlinks? #3300
• Decide on whether @embed support should need explicit enabling (via a flag or similar) or not: Embed proposal #3264 (comment)
• When to use file embedding and when not to use file embedding? Embed proposal #3264 (comment)

[verdverm]

Does the @embed proposal support embedding directories? The proposal does not seem to indicate so.

I would expect that if the path resolved (by glob or specific) is a directory, all files are embedded recursively, probably with the

[myitcv]

Great question, thanks @verdverm (although it appears like the end of your question got cut off?)

Per our brief exchange on Discord on this. I assume this question comes with some experience of Go's embed, specifically:

If a pattern names a directory, all files in the subtree rooted at that directory are embedded (recursively), except that files with names beginning with ‘.’ or ‘_’ are excluded.

We did toy with the idea of something like @embed(dir=$path), however this raised the following points/questions:

• If we wanted to maintain consistency with the rule for @embed(glob=$pattern) (where the type needs to be specified if the extension in $pattern is not fully specified), this would imply that @embed(dir=$path) would always need to specify a type.
• This would suggest the need for a special type=auto (for some spelling of auto) which is an explicit indication that file extension should be used during decoding. Possibly with the ability to provide some sort of fallback/default, pseudo-code type=*auto|text
• If we do implement support for ** in glob patterns, what value does @embed(dir=$path) then provide?

No clear/definitive answers (and if you have thoughts please do share them), but on balance, certainly for an initial implementation, we decided against designing/implementing @embed(dir=$path) in favour of saying we would fast-follow with support for **.

To ask a question by way or reply: if we added support for ** (we link to the LSP spec definition as the one we would likely use), would that satisfy your use case? Because with that support you would be able to do @embed(glob=**/*, type=text) or similar.

[verdverm]

Yes, I think double-star would likely cover the case

Fallback to text (the type=*auto|text) sounds great

[myitcv]

Yes, I think double-star would likely cover the case

Great.

Fallback to text (the type=*auto|text) sounds great

I've updated the top-level description capturing this feedback and "next steps".

[mpvl]

Note that we deliberately did not support auto in the embed proposal. Also fallback is tricky. The problem is that this behavior may change over time. Suppose your fallback is text. Right now, TOML files would be encoded as text. If at a later point TOML is supported, it would decode differently.

One way around this would be to, besides **, also support alternatives. This would allow **/*.(json|yaml), etc. This is one of the main reasons why we proposed to use the LSP glob.

If one would need to support different "fallbacks", one could have one could combine a @embed(**/*.bin, type=binary), @embed(**/*.md, type=text)`, for instance.

Of course, another way to avoid this problem is to tie the interpretation to some version specification in the module. But this is a bit less transparent, and it would be great if we could avoid that.

[verdverm]

Can multiple @embed() be merged together?

@embed(foo/*/test)
@embed(bar/*/test)
@embed(bar/baz/*/test)

[myitcv]

From my recollections of the proposal, this should be supported. And I'm almost 100% sure that I tested it working. And yet when I came to post a reply here I ran into #3294. So something in my recollection or testing is not working.

[myitcv]

Ah, this is what I did before (thanks to @rogpeppe for flagging):

env CUE_EXPERIMENT=embed
exec cue export x.cue
cmp stdout stdout.golden

-- x.cue --
@extern(embed)

package x

f: _ @embed(glob=*.json)
f: _ @embed(glob=*.yaml)

-- x.json --
5

-- x.yaml --
test

-- stdout.golden --
{
    "f": {
        "x.json": 5,
        "x.yaml": "test"
    }
}

This passes. I'll leave #3294 open to consider how to handle when multiple "instances" of an attribute as declared against a single field declaration.

[verdverm]

I wanted to show multiple directories in my example, the last 2 are interesting because one is a subdir of the other, want to make sure things don't get replaced

[myitcv]

I wanted to show multiple directories in my example, the last 2 are interesting because one is a subdir of the other, want to make sure things don't get replaced

This should work. Embedding is a purely syntactic operation. Any conflicts that arise because of contradicting @embed attributes will be surfaced by the unification that happens in that field.

[mpvl]

Yes, see also my previous comment. I think it would be quite strange to not support this.

[nxcc]

When loading files from a path, e.g. @embed(glob="le-cert/tls/tls.*", type=text), it would be beneficial to add a functionality to @embed that allows stripping a specified number of leading path components from the loaded file names, similar to the --strip-components option of tar. This would be particularly useful for scenarios where only the filename or a portion of the path is relevant. In this case I only want e.g. tls.crt as key.

Maybe by default only the basename should be used as the key? At least in all of my use cases, only the file name is relevant, never the full path.

[hansbogert]

What about situations where you don't actually care about the filename at all? I'd much rather get control and being able to reuse key/values from the imported file, much like you're able to do with cue import -l some.sub.key. If we could generalize a solution to cater to this as well, we pretty much have all flexibility you need.

[mpvl]

Note that one could always load a glob into a hidden field (e.g. _foo), and then use a comprehension to rewrite the map as desired. The path package has functionality to aid with that.

@hansbogert: I though about such cases as well. In my view, the tradeoff is between a slight increase in verbosity of using a comprehension, versus having to learn new syntax for such mapping. Since the comprehension possibility already exists, not adding options for this both avoids work and introducing a different way to do the same thing. If this is really a show stopper, we could still consider adding the syntax for this later.

[mpvl]

Here is an example:

_foo: _ @embed(glob=*.json)
for x in _foo { foo: (x.name): x } 

versus

foo: _ @embed(glob=*.json, key=name)

The latter is less verbose, but arguably not as clear. You will have to know/learn that key should be interpreted as an expression that is evaluated within an interpreted JSON file, which is then assigned as its key in the map. The comprehension syntax already exists.

[hansbogert]

@mpvl Yeah maybe relying on known mechanisms like comprehension is better. There is one thing that remains which could be an argument: performance. With the comprehension you'll have to loop twice, once for the loading of the non-cue files and second for the comprehension.

As someone whose never looked at the evaluator I've no idea if performance or memory size of using an extra round of comprehension matters in the grand scheme of things. However, I imagine the embedding will be popular with large kubernetes config repos where it might matter. You are probably the right person to reason about this if this matters, but I'm eager to know what that reasoning will be, just to understand cue better.

[mpvl]

I doubt the performance will matter in this case. I would expect the IO ops to far outweigh the constant per-file overhead for such an additional comprehension.

[jpluscplusm]

Naming

I feel the name used for this feature ("embedding") and the attribute name (@embed()) will cause confusion for some users.

The parallel with Go's embed directive is, I assume, intentional. However, the mode of operation is subtly but meaningfully different with CUE. Because CUE has no equivalent of a compilation step, the referenced files aren't "embedded" in the same way that Go embeds data. Here, because the result of an evaluation can now vary without modification of any CUE files or the cue command that's invoked, into what "thing" can the files be said to have been "embedded"? Embedding implies some level of fixed-ness, and that appears to be missing here. (Missing by design, I should say - this isn't a criticism of the lack of a compilation / "fixing" step!)

I think there will be cognitive dissonance between folks' understanding of what embedding already "means", coming from Go, and what's actually meant by CUE. And also for non-Go users, who I believe will expect some concept of fixed-ness when encountering the term "embedded".

As a possible alternative, I think @read() and "Reading" would accurately communicate the feature's behaviour. Additionally, I believe I recall some talk on Slack or elsewhere about the "emit" side in this feature space: emitting multiple files without requiring multiple cue export invocations. That could nicely pair with the term/attribute/name @write() and "Writing", in a way which, to my eye, would clearly indicate that the features were time-of-invocation-based (i.e. no "fixing in place" has happened) and would align with an existing plain English reading of both terms.

[FWIW, I don't feel this is pure bikeshedding; given that the feature's name will be baked in, on its first release, I think avoiding user confusion ahead of that release is important!]

[mpvl]

I also don't like the ambiguity arising from @embed. Very much open to new names. I think import matches semantics the best, but that may give confusion w.r.t. the import clause. Maybe that is less of an issue than embedding.

@include is another possibility, but that traditionally implies a verbatim inclusion, which this is not. Arguably the same holds for embed.

[hansbogert]

this might be a bit too tangential and/or might already have been taken into consideration internally, but why not use dedicated attributes for the file types? E.g., @json(...) and @yaml(...) , etc.

[jpluscplusm]

Apologies for the delay in replying here. I have an idea for a name which I think ticks all the boxes mentioned in this thread :-)

How about eval? Here's why I think it works:

• It accurately communicates the intent. The file/s pointed towards will be "evaluated" in the same way that the cue command would do -- interpreting data as data, text as text, structured data as structured data and so on.
• It has a pleasing symmetry with "emit", which could serve as the multi-file-export tag for some future feature
• It deliberately uses the same term commonly used by general purpose programming languages when they're executing something that's (quite often) dynamically specified. In such languages, eval is often one of the more dangerous functions, as it cedes execution control to something that's likely not an integral part of the program. Here, implicitly drawing on that parallel for programmers would be useful: it would communicate to those users that they're ceding control of their data specification to something that's not inherently part of the CUE they've defined. For non-programmers without that previous context, the lack of awareness of that parallel wouldn't be disqualifying - eval would simply be the name of the tag used by CUE for this feature.

[jpluscplusm]

I also recognise that my suggestion of @eval() cannot imply that the name of the feature should be "evaluating", as that collides too readily with other CUE uses of that term.

Continuing this thread's aim of finding alternatives for the term "embedding", I'd like to suggest the feature should be renamed "include"/"including".

For example, used in context on https://cuelang.org/docs/howto/embed-files-in-cue-evaluation/:

• The URL path .../howto/embed-files-in-cue-evaluation/ would become .../howto/include-files-in-cue-evaluation/

• The title "Embedding files in a CUE evaluation" would become "Including files in a CUE evaluation"

• The heading "Embedding files in an evaluation" would become "Including files in an evaluation"

• The sentence

  The @embed() attribute is designed to simplify workflows that require data to be loaded at evaluation time

  could become

  The @eval() attribute is designed to simplify workflows that require data to be included at evaluation time

• The sentence

  Include the @extern(embed) directive at the top of each CUE file that the @embed() attribute ...

  could become

  Place the @extern(include) directive at the top of each CUE file that the @eval() attribute ...

[rogpeppe]

My 2¢:

I don't like eval as a name. To my mind, "eval" is what the interpreter does when it evaluates a CUE program. This directive doesn't itself evaluate anything. Instead, it includes (embeds) some file contents inside the CUE program, ready to be evaluated. Also, there is no requirement that the data actually is evaluated. I believe it would potentially be OK for an evaluator to evaluate embeddings lazily on demand, and I can see that being very useful when numerous large data files are embedded but the expression I wish to evaluate only needs access to a small number of them.

"import" (as suggested by Marcel above) seems better to me, but in my view that is even more confusing than "embed" because of the somewhat-overlapping use cases between importing packages vs files. Saying @extern(import) would seem a bit weird when it appears just above an actual import directive that's importing external packages.

I honestly don't think that there is a serious issue with the naming overlap between the language feature that's called embedding (which doesn't have any associated keywords, and is always referred to as "embedding" never "embed") and the term "embed" as used for this feature. Note that the technical ambiguity doesn't seem to be a problem in Go, which has both file embedding (//go:embed) and struct embedding.

Another possibility for a name might be "include". This feature isn't a million miles away from C's #include directive, although it's clearly more principled than that. But "include" is close enough to "import" that I think I generally prefer "embed".

Finally, I don't see this as a dangerous feature. The design is careful to avoid anything which involves reading files outside of the main module or invoking anything non-hermetic. I personally have been using the feature as convenience to allow myself the choice of representation within a CUE program. It works great for that, and it doesn't feel like I'm doing anything unprincipled or dangerous.

[jpluscplusm]

Operator Control

Currently, the cue operator is in charge of how and if the CUE hermetic seal is broken. This proposal changes that stance.

The person who invokes a cue command currently has direct control of which command line flags are present, and knows that if -t or -T are included then CUE's hermetic seal has been broken - with all the potential downstream implications. This proposal changes that relationship between CUE author and CUE operator, allowing the author to include system-local files in an evaluation, without the knowledge or involvement of the operator.

I feel that the relationship between author and operator should be maintained in its current state, and that this feature should be enabled via an explicit command line flag. I acknowledge that this would take the "just trying out the feature" process from 2 elements (@extern(embed) and @embed()) to 3 (as before, but with --allow-file-embedding or equivalent), but that feels like the correct operator-respecting choice, to me. The alternative would seem to be having to weaken any rhetoric about CUE's hermetic seal, as any given CUE invocation could, given the right directory setup, involve the contents of unexpected files - leading to the output of any invocations that process user-provided (i.e. potentially hostile) content having to be reviewed by the operator after any given invocation.

[myitcv]

Thanks again for the feedback and thoughts here, @jpluscplusm. We thought hard about exactly this point and landed on the proposal as it stands based on the following reasoning:

• What is proposed here is a very common operation; requiring a flag for a very common operation is annoying.
• The .cue files that @embed things, and the files that are the target of the @embed, are known statically. We will expose this via https://pkg.go.dev/cuelang.org/go/cue/build#Instance, and likely some cue list command a la go list.
• From a security perspective, the files that can be the target of @embed are limited to those contained by the module that contains the .cue file that is doing the @embed. Furthermore, the code that executes during @embed is just the same as would run during cue import.
• Taking the two previous points together, files that are part of @embed can be considered to be hermetic in just the same way that any files participating in a CUE evaluation are: they just have a slightly different source.

any given CUE invocation could, given the right directory setup, involve the contents of unexpected files - leading to the output of any invocations that process user-provided (i.e. potentially hostile) content having to be reviewed by the operator after any given invocation.

Given the comments above, what sort of situation are you envisaging here?

Stepping back somewhat, if I have understood the general point you are making (and please correct me if I am wrong), its main thrust is the decision on whether embedding should need to be explicitly enabled or not, as opposed to any objection to @embed per se. Points contributing to a "conclusion" on that explicit vs implicit flag include:

1. How common is the use of @embed?
2. How "surprising" it would be for files that are the target of @embed to be read without a flag/equivalent?
3. How safe is @embed to be used without explicit "enabling"?
4. Do we fundamentally alter the nature of CUE with such a feature?

Hopefully I have, in the course of this answer but also #3264 (reply in thread), provided the rationale behind how we landed on the proposal as it currently stands: we see it as a common operation, that can easily be documented and explained (acknowledging we are living in a world pre-@embed today so there will be an element of surprise in any transition), and the hermetic nature of CUE is left intact because @embed is fundamentally a shortcut to having to run a number of other operations.

That's not to say we landed on the right answer, rather just me sharing our thought process if you will!

Would appreciate more detail on the attack vector you think exists because that clearly gets to the heart of point 3.

[rogpeppe]

@jpluscplusm

allowing the author to include system-local files in an evaluation, without the knowledge or involvement of the operator.

An important caveat here is that it's not possible to include any system-local files, but only files within the same module.
To me this feels very similar to the way we are able to import other packages within the same module: there's no lack of hermeticity there, just a level of indirection.

To be clear, I see @embed as only syntactic sugar for including the contents of a file within the module as a value in the current evaluation. If the file happens to vary from invocation to invocation that's really no different from mutating a CUE file within the module AFAICS.

A crucial point here is that an external CUE dependency does not have the ability to read system files, or any files outside of that dependency's module. So there's no lack-of-hermeticity from that direction AFAICS.

If the same-module restriction were dropped or loosened, then I think your concerns would be fully justified, but currently I don't see there's a significant issue here.

[infogulch]

The key here is that everything in the cue module directory is considered part of the cue module, not just the files that end in .cue. With this mindset, it's no problem if you can read any file in the module.

[jpluscplusm]

In the absence of a --allow-embedding flag, I feel there should be a --disallow-embedding flag that disables the feature. This will allow operators who are working in less tightly controlled setups to invoke the cue command and know that only CUE packages and data files they explicitly mention will impact the evaluation, with the same level of surety that they have currently. I feel it would also be beneficial if there were a envvar whose setting were an equivalent of the flag's effect - be that an explicitly named, single purpose envvar, or a generic CUE_OPTIONS var, etc.

[mvdan]

@jpluscplusm just so that I understand your point of view better - how is @embed(file=some-untrusted-source.json) being present in a CUE file any different than import "some-untrusted-source.com/foo"? Both result in loading some amount of external files which you did not give explicitly, such as from cue export .. From what I can see, having no option to disable embedding is akin to having no option to disable importing packages.

Note that it is not possible to embed files outside of the current module, so I can't see how you should be worried about embedding any files.

[myitcv]

When to use file embedding and when not to use file embedding?

As a follow-up to the discussion in #3264 (comment), we should make clear when to use file embedding and when not. For example, when writing a cue cmd workflow, the correct way to load a file is via tool/file.Read. To use file embedding in this case would render it impossible for the workflow in question to be reused elsewhere.

There are perhaps similar/further FAQs which could be answered in the proposal, or as a follow-up documentation on https://cuelang.org/.

[DavidGamba]

I have a very simple task that generates a set of YAML documents for me:

package workflow

import (
	"encoding/yaml"
	"tool/cli"
)

objects: [for v in objectSets for x in v {x}]

objectSets: [
	workflow,
]

command: dump: {
	task: print: cli.Print & {
		text: yaml.MarshalStream(objects)
	}
}

I wouldn't want to convert this to use some form of tool/file.Read when a cue eval works from the same directory.

[DavidGamba]

FYI: this is allowed now, hopefully the proposal doesn't remove support for this.
See #3549 (comment)

[DavidGamba]

Other than slack I couldn't find documentation on how to enable the embed experiment from the Go API.
Maybe the Go guide should be updated to indicate how to enable experimental features.

For others looking for this:

import (
	"cuelang.org/go/cue/cuecontext"
	"cuelang.org/go/cue/interpreter/embed"
)

	os.Setenv("CUE_EXPERIMENT", "embed")

	embedding := cuecontext.Interpreter(embed.New())
	ctxOpts := []cuecontext.Option{embedding}
	c := cuecontext.New(ctxOpts...)