Deprecate SNI endpoints and figure out a replacement

[mithrandi]

The endpoints rely on the tls-sni-01 challenge to operate, which is dead. We should deprecate them ASAP to avoid confusing people switch them to use tls-alpn (#136), and figure out some replacement (probably using http-01) that is as easy to use. (Unfortunately I'm not sure this is possible 😿 )

[njsmith]

I guess you could buffer incoming data long enough to see whether it looks like GET /.well-known/acme-challenge/...?

[mithrandi]

Wouldn't actually work because we're listening on the wrong port (we're listening on something that gets port 443 traffic, but we need port 80 traffic). I suppose we could just open up a new port, but the syntax to specify it would be awful.

[mithrandi]

It looks like tls-sni will live on in tls-sni-03, so we don't necessarily need to kill it, but we should document prominently that it won't work for new users at the moment.

[glyph]

The syntax can be an exercise for the reader here, but endpoints aren't necessarily the wrong way to do this. We just need an endpoint that listens on 2 underlying ports; one HTTP + redirect + .well-known, and one HTTPS.

[GaretJax]

According to https://tools.ietf.org/html/draft-ietf-acme-acme-01#page-37 (but I don’t know if Let’s Encrypt actually does it), https can also be used for validation, in which case the certificate is ignored.

[warner]

I just stumbled into this, spent an hour debugging until I remembered that tls-sni-01 had been disabled. Would you mind a PR that adds a note to the docs, warning that the basic endpoints probably won't work for new sites, pointing at this issue?

I wound up building the 80+443 site that Glyph mentioned (except without the redirect). I'll polish it up and see if it might work as a different form of endpoint descriptor. The tls-sni-01 challenges must be handled on port 80, so maybe we could force that first endpoint to be tcp:80 (although if you're doing port-forwarding from some other system, maybe not).

At the very least it'd be nice to have a single function/constructor to build these endpoints. The code I wrote had to import a lot of details from txacme, which I'd rather hide from applications.

[mithrandi]

+1 to both.

[glyph]

@warner anything ever come of this?

[warner]

Not yet.. lemme see what I can get done this weekend.

[warner]

Ok, here's what I'm thinking:

We'll want an Endpoint class, and a parseable serverFromString format. The class can have more features than the string (if you need more control, your program has to build the new Endpoint itself, rather than using serverFromString).

The Endpoint will own the http-01 responder (which must be reachable by the target domain name, on port 80, speaking plain HTTP, and serving a .well-known/acme-challenge document). This will probably be listening on the real local port 80, but port-forwarding is a thing, so that should be configurable (the class can accept a whole endpoint, with interfaces= too, but the default should be tcp:80).

The most common use case is for HTTP, so by default the Endpoint should also redirect everything that isn't .well-known/acme-challenge to the same path on a parallel URL with https. I'm only 99% sure this can be done without knowing the hostname. I can imagine non-HTTPS use cases (SMTP comes to mind) so we might consider an option to disable this forwarding. If you don't forward everything, then you might want to populate the HTTP site with something, so it should be accessible.

The Endpoint needs a place to hold its certs, as usual.

The Endpoint also owns the txsni endpoint that shares the certDir with the ACME client/responder. This txsni endpoint itself creates a real TCP endpoint that accepts the TLS connections. The txsni parser lets you include arbitrary kwargs (like interface=) for the TLS endpoint, so maybe we should too.

However it'd be nice to allow some future kwargs for the ACME process (personally I'm eager to see #112, maybe as hostnames=X,Y), and we don't have a good way to assign kwargs to the right parser. So I'm inclined to not make it completely general, but instead provide specific kwargs for the ones that we know are useful (just interface=), leaving future keys for expansion of the ACME or txsni parts.

The Endpoint doesn't need to run the HTTPS server, that's up to the caller. But it does need to run the HTTP server.

So how about:

def MumbleAcmeHTTP01Endpoint(reactor, certPath, tlsEndpoint=None,
                             responderEndpoint=None, forwardHTTP=True):
    assert IFilePath.providedBy(certPath)
    if tlsEndpoint is None:
        tlsEndpoint = serverFromString("tcp:443")
    if responderEndpoint is None:
        responderEndpoint = serverFromString("tcp:80") 
    txsniEndpoint = txsni.tlsendpoint.TLSEndpoint(tlsEndpoint, SNIMap(..))
    ...
    return acmeEndpoint

# non-redirecting usage:
ep = MumbleAcmeHTTP01Endpoint(reactor, certPath, forwardHTTP=False)
ep.httpSite # not sure what this is good for, but available
ep.httpRoot.putChild(b"", static.File(root_document))
ep.httpRoot.getStaticEntity(".well-known").putChild(b"other-thing", thing)

and an endpoint syntax of:

acme-http-01:certDir[:ARGS]

Where the optional arguments (with examples) are:

• http-port=80
• http-interface=127.0.0.1
• no-http-redirect (note: does not have =, does not take a value)
• tls-port=443
• tls-interface=127.0.0.1

My primary uncertainties:

• spelling of the acme-http-01 string prefix
• spelling of the endpoint class name
• use http-redirect=false, so all optional args take values

Is there some clever way to allow the tls-port/tls-interface values be passed in as a normal string? Maybe we declare that some special argument name terminates the ACME arguments and everything after that gets reparsed by serverFromString for the TLS endpoint? So like acme-http-01:http-port=8080:endpoint:tcp:10443:interface=127.0.0.1 ? That's obviously not composeable: we can only have one such marker in any given endpoint string, but it's probably more important to let you control the final TLS endpoint details than accomodating new arguments for the HTTP endpoint. And it'd retain our ability to do something like acme-http-01:hostnames=example.com,www.example.com:endpoint:tcp:10443.

[glyph]

This all sounds encouraging as a spec, very much in line with my own thinking. If you’ve got the energy for a PR, I think that is the next step:)

[warner]

yep, I'll take it

[mithrandi]

That all sounds good, but I have one concern:

So I'm inclined to not make it completely general, but instead provide specific kwargs for the ones that we know are useful (just interface=), leaving future keys for expansion of the ACME or txsni parts.

A lot of useful endpoints to listen on will take weird arguments, I think; for example, systemd, or tor. By saying "the ones that we know are useful" what you're really saying is "only tcp can be used" which seems unfortunate. A suggestion made elsewhere is that we name our own arguments like txacme-blah which should avoid most namespace collisions at the cost of some extra typing.

[warner]

Good point, I agree we should do whatever we can to enable generality of the string form, and I'm happy to have the txacme-specific arguments take a prefix to avoid collisions.

The problem I don't know how to solve is that we've got two separate endpoints that could both be receiving arbitrary future-extension arguments. The "responder endpoint" (which must be visible at port 80 from the LetsEncrypt challenge servers, but of course this could be routed through systemd or twisted.web.distrib or something) is one, and the actual "TLS endpoint" (managed by txsni) is the other.

Huh, what if we said that every responder-endpoint-XYZ= argument was converted into an XYZ= argument and passed into the responder endpoint serverFromString() call? And likewise for the tls-endpoint-XYZ= values? And we could also honor the same mapping for no-value args (e.g. tls-endpoint-ABC maps to plain ABC). Is that too wordy? You couldn't build up the txacme endpoint string by simple concatenation of the original endpoint string, but it'd provide access to any possible argument name for both internal endpoints.

[glyph]

99% of the time I think you'll want to use the defaults, so a wordy escaping strategy is totally fine with me. As long as txacme:certificates:4443:8080 (or ... something) for easily tweaking port number & cert dir (the only things most people will need to configure) doesn't require anything novel-length.

[glyph]

The major place I could see actually wanting something sophisticated would be in combination with the proxy: endpoint, so that might make a good use case for discussion.

[mithrandi]

I don't know if anyone is actually using txacme with systemd port activation or txtorcon, but those seem like obvious use cases that I'd really like to support easily if it's not going to make the whole thing awful. OTOH it's not worth it if the "easy" endpoint ends up being hard.

[warner]

https://github.com/warner/txacme/blob/129-http01-endpoint/src/txacme/endpoint_http01.py is my first cut. Not ready for a PR yet (no tests, no parser, and I haven't even tried to run it), but feedback still welcome.

[mithrandi]

LGTM. I have some potential comments on some minor issues, but I'll save those for when the branch is actually complete, since no doubt the code will change some in that process anyway; the overall design seems good though. I notice you would benefit from #108 being resolved.

[twopir]

I've started getting NoSupportedChallenge errors for TLS-SNI. When is ALPN or HTTP-01 expected to be supported?

[mithrandi]

Unfortunately nobody seems to be actively working on these, but see #136 .