Subject: Re: [saag] ASN.1 vs. DER Encoding

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 2 Apr 2019 22:05:04 -0500

On Sat, Mar 30, 2019 at 05:25:35PM -0500, Nico Williams wrote:
> On Sat, Mar 30, 2019 at 10:31:02AM -0500, Benjamin Kaduk wrote:
> > On Wed, Mar 27, 2019 at 10:15:46AM -0500, Nico Williams wrote:
> > > On Wed, Mar 27, 2019 at 09:45:16AM +0000, Peter Gutmann wrote:
> > > > >Thus there is almost zero benefit to self-describing encodings.
> > > > 
> > > > ... apart from the fact that they can be statically analysed to check whether
> > > > they're well-formed or not, unlike the encodings in PGP, TLS, IPsec, SSH, ...
> > > 
> > > The protocols you list don't use a formal syntax, which instantly makes
> > > validity checking harder (can't generate the code!).  But if they had
> > > used XDR, or ASN.1 with PER/OER/..., you could in fact automatically
> > > check the validity of the encoding of a message.
> > 
> > N.b. that the protocol descriptions in RFC 8446 were run through an
> > automated syntax checker (IIRC, by Kazuho, though I'm not confident of that
> > and only bought 20MB of data for this  plane flight).  So I'm not entirely
> > convinced that this claim applies to TLS 1.3.
> 
> ISTR hearing about that, but the RFC does say:
> 
>   3.  Presentation Language
> 
>      This document deals with the formatting of data in an external
>      representation.  The following very basic and somewhat casually
>      defined presentation syntax will be used.
> 
> which is basically saying it's not a formal syntax.

We never bothered to remove the disclaimer, but the meaning is well-defined
and that's really all you need to make it a "formal syntax".  That text
seems unchanged from RFC 5246 (which really was a bit more slapdash about
the formal definition).

> Now, perhaps it can be be formalized.  If so, was that done for the
> checking you mentioned, and was it lost?  That would be unfortunate.
> 
> Are there implementations that generate codecs for TLS?

It looks like (per https://github.com/tlswg/tls13-spec/pull/999) I was
thinking of Kazu Yamamoto's
https://github.com/kazu-yamamoto/tls13-spec-validator which I guess will
generate haskell for the presentation language.
https://github.com/tlswg/tlswg-wiki/blob/master/IMPLEMENTATIONS.md does
list a Haskell implementation with Kazu as author (if we chase the link).
But I don't know if the implementation uses the generated code directly or
not.

Kazu, can you enlighten us?

Thanks,

Ben

---

From: Nico Williams <[email-masked]>
Date: Thu, 4 Apr 2019 10:54:20 -0500

> seems unchanged from RFC 5246 (which really was a bit more slapdash about

Good.  An informal syntax can be good enough indeed.  And the use of a
formal syntax can be insufficient (e.g., Heimdal's ASN.1 compiler has
support for non-standard ASN.1 usage in RFC1510.)

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 22 Apr 2019 16:47:24 -0400

On Sun, Mar 31, 2019 at 7:36 AM Stephen Farrell <[email-masked]>
wrote:

> On 31/03/2019 08:39, Carsten Bormann wrote:
> > There is no free lunch in this space.
>
> Oddly, there sort-of is... if one ignores the ~every-5-years
> debate that the new-way (asn.1/der,asn.1/<foo>,xml+dtd,xml+
> schema,json,cbor...) is obviously the right answer to everything,
> and just go eat your lunch... then that's nearly a free lunch:-)
>
> I never understood how so many people get so excited by this
> topic myself. Seems to me programmers will always find a way
> to do as well or badly as ever despite the differences in this
> kind of tooling. (Training, experience and other kinds of
> tools like more strict compiler options can make a difference.)
>

Only five? Seems like every six months.

Yes, of course write a compiler to parse any data structure, especially IP
packets. I can guarantee you that IPv6 packet exploits will be a recurring
source of software vulnerabilities for decades into the future.

I have written several ASN.1 encoders and decoders over the years. My take:

1) I never touch ASN.1 unless I am being paid to do so and in advance. Nor
would I expect anyone else to do so.

2) ASN.1 is a very good idea in principle that is marred with some very
unfortunate practice that render it worse than useless.

3) Having six encoding rules is a defect, not a strength. The point of
standards is to remove choices that don't really matter. Having six
incompatible encodings makes it matter.

4) The DER encoding rules are botched. They require use of nested length
delimited fields for a start which is inherently insecure unless the
receiver checks every inner structure to make sure that it is correctly
nested inside the outer.

5) The botched DER encoding is even worse because the nested lengths is the
use of nested variable length length specifiers. This means that the only
efficient strategy for encoding ASN.1 is to fill the buffer in the REVERSE
order starting with the last item.

6) Since there is no advantage of using ASN.1 over other choices and you
will get three years of gripes and debates over schema peculiarities, the
only use I can find for ASN.1 is for the small number of legacy
applications that require it. The only one that I have not found to be
unavoidable being PKIX which requires the worst-of-the-worst DER encoding.

7) To add insult to injury, the canonicalization that DER encoding purports
to provide is of absolutely no value. If I am checking the signature of
octet sequence X, absolutely harm should come from the possibility that
there exist an octet sequence Y that has identical semantics which could
have been signed instead. Yes, I can construct ludicrous scenarios in which
that would be true but any system that relied on canonicalization is broken.

8) PKIX requires DER but does not provide a canonical form for a
certificate. You cannot strip an X.509 certificate to its elements, put
those elements into an X.500 directory and then reassemble the cert even if
you want to because the extensions are specified as a list, not a set.
Given that 1TB of SSD currently costs $125 and certificates are a few KB at
most, this would be a very silly thing to want to try in any case even if
there were any people still operating X.500 directories.

---

From: Nico Williams <[email-masked]>
Date: Mon, 22 Apr 2019 16:14:50 -0500

> incompatible encodings makes it matter.

Six?  I believe there's more.  Ever heard of GSER?

BER, DER, CER, PER, OER, XER, JER(?), GSER, and who knows what else.

You could easily make encoding rules for ASN.1 out of XDR, NDR, CBOR,
and many others.

It's not a defect of the spec.  It's a defect of history.

In 1984 the ITU-T crowd loved themselfs some TLV encodings.

The IETF crowd did not.  Thus PER.  But PER never took off because the
specs weren't free at the time, and there was no tooling.  TLVs at least
looked easy.

Meanwhile the IETF loved itself some text-based protocols -- lots of
them.  And while neither XML nor JSON came from the IETF, they found a
home here.

It turns out there's a great deal of demand for both, a) "textual", and
b) binary encodings.  And for (b), TLV kinda sucks, but it's what we've
done for 30 years in many cases.  We also have "bits on the wire"
encodings.

That's why we have all the ASN.1 encoding rules, XDR, NDR, XML,
FastInfoSet, JSON, a whole bunch of "binary JSON" formats (including
ones not brought to the IETF, such as PostgreSQL's JSONB, which is
actually quite neat), and now all the *buffers rules (protocolbuffers,
flatbuffers), and who knows what else I might be missing.  Oh, and
things like MIME, which in fact are encoding rules.  And SSHv2, and TLS,
and...  And S-expressions, and Perl5 data dumper, and...

Easy, safe prediction: we'll have more encoding rules soon.

> receiver checks every inner structure to make sure that it is correctly

Yes.  The decisions made in both DER and CER make both of them difficult
to use in actuality.  The ITU-T could have made better choices for a
canonical flavor of BER :(

> order starting with the last item.

It's worse: you have to realloc as you go, or build a rope, or do one
pass to compute length and one to encode.  DER is truly awful.

> that would be true but any system that relied on canonicalization is broken.

Agreed.  And yet even now we find people who want a canonical JSON :( :(

> there were any people still operating X.500 directories.

One thing we've learned is that we *don't* want directories or white
pages of any kind, not outside the corporate environment.

And x.400/x.500 naming is an awful disaster.

Another thing is that the specific form a certificate takes is not
terribly interesting, but the names and issuer are.  And even the issuer
wouldn't be that interesting in a properly hierarchical PKI, but we
never got one (except for DNSSEC).

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 22 Apr 2019 19:15:14 -0400

> done for 30 years in many cases.  We also have "bits on the wire"

Hence the reason I decided to smoosh in binary encodings into JSON to
create a superset so that one decoder can read either the JSON or the
binary version.

> Easy, safe prediction: we'll have more encoding rules soon.

True. But here is the thing, I have a series of schema driven parsers that
use essentially the same schema language to create parse r for ASN.1, XML,
JSON, DNS Records, TLS, RFC822, RFC821 etc. I can throw out YANG, SML
Schema, Relax etc. if needs be as well.

The basic principle is to allow the basic encoding to be specified (e.g.
integer) plus hints if needed in addition (e.g. 16bit, 32bit, signed,
Little, Big endian, etc.)

> canonical flavor of BER :(

Had they chosen to require lists, sets, etc. to use the indefinite length
encoding instead, the problems would have gone away. But nobody got round
to implementing until after they made that choice.

> pass to compute length and one to encode.  DER is truly awful.

I use the rope trick.

> pages of any kind, not outside the corporate environment.

I disagree. There is a role for a personal directory mapping local names to
devices and users. It just looks nothing like the nonsense of X.500 and
LDAP.

If I say 'close garage door', it is implicit that it is the one connected
to the house I am currently in. And I want my Apple, Google, Microsoft,
DEC, etc. devices to all make use of the same local names in the same way
which mean that they have to be MY local names from MY directory, not
whatever sludge the AI heuristics driving Siri, Alexa, Hal, etc. try to
infer.

Basically, if we want to sell consumers hundreds of IoT devices then we
have to either admit that all that stuff we sold to enterprise customers
was unnecessary and they never needed it or decide it is needed and find
some way to provide the same sorts of functionalities but without the make
work that enterprise software sales require.

---

From: Russ Housley <[email-masked]>
Date: Mon, 22 Apr 2019 19:54:52 -0400

> And x.400/x.500 naming is an awful disaster.

They are not the same.  Once can completely avoid X.400 names, but the X.500 one are used in certificates.  I strongly encourage people to keep it simple.  The bits on the wire sitll get too complicated, but the code can mostly do exact match processing.

Russ

---

From: Nico Williams <[email-masked]>
Date: Mon, 22 Apr 2019 22:54:16 -0500

On Mon, Apr 22, 2019 at 07:54:52PM -0400, Russ Housley wrote:
> They are not the same.  Once can completely avoid X.400 names, but the

They are not, but they are similar, and similarly difficult to use.

> X.500 one are used in certificates.  I strongly encourage people to
> keep it simple.  The bits on the wire sitll get too complicated, but
> the code can mostly do exact match processing.

To keep it simple means to leave the subjectName empty and use dNSName
and rfc822Name SANs instead wherever possible.

Naming is more than half the battle.  Internet-style naming of things
won long, long ago.  It's not just that users can handle domainnames and
name@domainname syntax but not x.500, but that x.500 naming is
fiendishly difficult to handle in code, or even in specs -- there's not
even a lossless textual representation of x.500 names [RFC4514]!

Nico
--

---

From: Watson Ladd <[email-masked]>
Date: Mon, 22 Apr 2019 21:02:54 -0700

> even a lossless textual representation of x.500 names [RFC4514]!

Let us not forget the valiant battle to enforce the requirements that
CAs know that Bremerhaven is in Bremen and not in Niedersachsen. I
await the discovery that a small company in Baarle-Hertzog is actually
three feet across the border, with dire consequences for the web PKI.
(If the border moves and you don't, does the certificate need
revocation? Or what about countries being renamed?)
>
> Nico
> --
>
> _______________________________________________
> saag mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/saag

-- 
"Man is born free, but everywhere he is in chains".
--Rousseau.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 23 Apr 2019 01:08:11 -0400

> revocation? Or what about countries being renamed?)

The only circumstance in which geography is usually significant in the
WebPKI is when the certificate is EV or OV which are supposed to provide a
degree of accountability.

If you register a business in Germany, or for that matter you merely
purport to hold said registration, you are accepting a certain degree of
accountability under German law.

If you are using a DV certificate and selling fraudulent goods, you may
have customers in 100 different countries, none of which consider you to be
their problem in particular.

If you are using an EV certificate and it says Germany, well we certainly
have a nexus in that particular country and their police are more inclined
to investigate the fraud and hold the perpetrators accountable if possible.

The original design brief for the WebPKI was limited to making online
commerce no less safe than bricks and mortar stores.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 23 Apr 2019 11:08:23 -0400

> and rfc822Name SANs instead wherever possible.

Yes, but we can't leave the IssuerDN empty, and if we want chains of
certificates (we do), then we need to put something into the subjectDN.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Nico Williams <[email-masked]>
Date: Tue, 23 Apr 2019 10:19:31 -0500

> certificates (we do), then we need to put something into the subjectDN.

Well, there is id-ce-issuerAltName, but indeed, the issuer Name must not
be empty.  At least we can encode domainnames as DNs, and there's no
need to represent, e.g., email addresses as issuer DNs.

In any case, issuer names don't leak as much into UIs, so it's less
critical that we use dNSName SANs for them.

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 23 Apr 2019 14:25:30 -0400

On Tue, Apr 23, 2019 at 10:19:31AM -0500, Nico Williams wrote:

> be empty.

Of course the chaining need not in principle have been based on a
fictional global X.509 directory tree.  It could have been just key
ids, with the CA names as commentary for human eyes and audit trails.
The only downside would then be loss of the ability to bypass path
length constraints via self-issued certificates.  Not clear we'd
really miss that.  But this is of course entirely hypothetical...

FWIW, despite clear non-compliance with RFC 5280 and potential
interoperability risk, some users seem to manage with "self-signed"
(below skid == akid) certificates that have empty DNs for *both*
the subject and the issuer (and indeed no SANs of any kind).

These are of course outside the WebPKI, used solely for unauthenticated
or DANE TLS.  A live example below, yes, in continuous use for the
last 5 years or so. [ The 4096-bit RSA key and ~1000 year validity
is a bold challenge to the coming scalable QC crypto apocalypse.
:-) ]

-- 
	Viktor.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            c3:26:2b:13:ca:b1:36:72
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: 
        Validity
            Not Before: Jul 27 14:59:59 2014 GMT
            Not After : Nov 27 14:59:59 3013 GMT
        Subject: 
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    00:b6:d3:42:35:68:e9:2a:9e:ba:f8:f0:f4:bf:30:
                    b5:0b:40:cd:10:4b:20:94:aa:fc:e8:d3:b1:b8:15:
                    cc:24:ba:7f:95:b5:85:92:e9:d5:97:70:d3:fd:b3:
                    c9:91:ba:d5:85:5d:c6:6d:98:8b:c3:b3:79:74:a7:
                    41:c6:f4:df:14:53:bb:90:21:72:71:ba:e2:56:03:
                    0a:0b:a9:db:d5:92:d3:90:58:4e:eb:a4:8b:51:80:
                    db:5f:56:26:cf:9b:26:a8:2e:42:df:54:14:86:4e:
                    1f:ad:b2:9c:57:54:16:7a:39:25:a3:b3:90:97:eb:
                    70:92:04:27:10:b6:fd:9e:70:4f:b2:02:e2:fa:6d:
                    90:eb:9a:0c:64:3c:31:86:4c:98:99:47:00:75:b6:
                    d0:bb:80:02:13:c7:43:97:24:ec:1e:3e:b1:1c:d6:
                    c7:b7:de:fc:e8:bb:c6:d8:20:74:16:09:27:2d:17:
                    17:a5:a4:41:d0:f6:60:de:a2:84:fa:e4:8d:dd:1e:
                    98:7e:19:75:a4:87:52:18:45:d9:6d:39:3e:2c:b2:
                    64:1a:13:37:26:3f:72:8c:7d:fe:2e:d6:26:d7:cc:
                    37:aa:06:4a:2f:ea:bc:0f:00:5f:d5:30:79:e8:11:
                    21:64:03:b9:91:e5:da:47:6b:7d:43:e6:5e:20:e8:
                    1d:1d:1e:3d:b8:57:62:01:98:13:5b:cc:a8:9f:6b:
                    d2:34:e0:6f:86:b8:ac:9d:89:f1:e9:27:b9:f8:55:
                    ce:a2:8a:33:2b:ac:3a:65:c0:fb:12:b8:f7:5a:47:
                    a6:ea:83:80:88:0f:ca:d4:d5:dc:62:5c:08:d9:cf:
                    e6:ca:fe:32:00:9e:e3:c0:53:99:21:a3:c9:4f:66:
                    07:fc:61:e2:20:18:01:7f:61:dd:e1:72:b5:fd:c3:
                    97:23:2a:51:bf:42:58:64:0d:2b:4e:cc:85:a0:5e:
                    01:52:2b:7b:46:f0:63:19:9b:a3:5e:2c:70:23:36:
                    a3:a9:3a:b3:60:2e:ad:78:68:96:ce:a4:4c:ea:13:
                    77:02:97:c4:55:82:f3:fd:3b:f3:f4:65:4e:dd:3b:
                    fe:d2:dd:d0:da:29:e8:3e:dd:a9:e3:c6:16:db:eb:
                    f8:90:72:dc:54:37:17:15:c9:43:1f:de:9d:5b:02:
                    5e:03:a9:3e:78:75:15:4d:bc:84:bf:a0:7e:4a:68:
                    7d:2b:c6:c5:b5:da:09:8b:f3:45:6e:82:2b:8b:be:
                    e9:5d:b7:b3:f0:e8:0d:04:8c:e3:b8:ca:23:1d:dc:
                    10:09:09:2e:1e:bf:23:4c:67:be:64:c1:90:fd:62:
                    57:17:d4:33:e6:1d:4c:70:d7:58:f6:17:5e:d2:4b:
                    d5:1f:9b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                98:C6:9B:D5:20:5C:1D:A8:31:39:BD:78:11:37:FF:BD:AD:5B:BD:59
            X509v3 Authority Key Identifier: 
                keyid:98:C6:9B:D5:20:5C:1D:A8:31:39:BD:78:11:37:FF:BD:AD:5B:BD:59

            X509v3 Basic Constraints: 
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         8d:47:1d:df:5f:63:ec:db:7b:a3:a3:a6:50:d0:76:f5:1a:86:
         da:21:bf:78:4d:4c:ab:ef:af:a1:be:e9:a5:29:20:6b:05:a3:
         88:85:0e:57:17:9c:e6:8c:f5:87:c7:07:a3:7b:ed:7d:f4:03:
         07:5a:6e:b4:bf:9c:db:6d:33:24:ae:4d:0e:39:06:54:9e:71:
         68:f6:5d:58:e9:19:ff:ef:e2:e5:7c:a9:b9:da:21:dd:14:19:
         d8:c1:6b:ab:ae:fd:2f:86:14:b9:8f:bf:77:75:b8:07:cc:0a:
         62:8a:00:98:c4:fb:0e:ec:ef:f7:11:88:0a:05:0e:ef:9b:c0:
         98:e0:39:47:c0:83:af:5a:f6:aa:3d:8f:2c:5d:b1:95:b4:93:
         a1:86:bf:1d:b1:45:91:e5:7f:6f:63:ab:59:cf:03:4e:c0:37:
         fe:ce:9f:2d:cd:64:a1:81:62:00:79:32:4d:b0:43:2e:58:6e:
         c7:79:f7:b6:74:be:c9:65:c6:2f:d0:e9:b8:56:60:d4:46:48:
         d8:6d:da:b2:81:59:a9:f4:94:8c:c4:9f:f6:ab:16:6f:f1:04:
         e7:e9:2a:bb:04:1f:4d:c5:c2:e0:0b:b0:60:d8:1c:31:59:da:
         c6:32:6c:77:8b:db:e7:77:88:4d:15:45:c9:ea:b8:95:5a:d3:
         d6:5f:19:ed:cd:5d:84:0d:30:75:70:ac:a3:9a:6d:83:fe:bc:
         60:fa:bb:2b:48:d7:12:eb:4a:e3:40:bf:01:56:a9:0d:d4:fc:
         49:88:70:6b:0a:24:36:e8:c2:dd:ea:6c:67:cf:5e:d2:0a:7a:
         31:b8:92:93:7c:f5:8c:91:8e:e9:d9:39:ec:1f:f2:98:0c:3d:
         d5:33:33:53:bd:b1:63:b6:18:e3:20:c6:50:2a:f1:09:50:5d:
         88:69:76:91:38:a1:c1:47:71:09:12:75:6d:a0:17:72:ad:e6:
         78:40:18:d3:04:04:70:3a:bf:74:45:0c:48:7a:7b:fe:0a:fd:
         ff:cb:ae:f7:85:50:fa:e2:23:73:87:54:ea:80:7e:c9:5f:da:
         80:3f:af:04:3a:58:d8:4b:24:75:58:a0:c5:94:0a:b8:8e:62:
         15:7e:3e:da:41:a8:a2:80:1b:c6:43:03:ae:2c:8c:fc:c7:83:
         df:38:df:b8:12:d2:ac:c1:10:b4:66:75:77:c8:a5:6f:49:16:
         c4:27:04:c2:fe:52:a4:ef:62:86:25:00:e7:ce:02:e7:4d:6c:
         c8:60:83:1f:4c:ba:d9:1b:83:da:cc:5d:bf:89:37:04:a7:85:
         62:de:4d:2c:4e:d0:13:c4:cd:81:51:4a:b0:07:53:95:6f:42:
         9e:2e:32:12:7b:1c:c1:c3
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----