Subject: Re: [saag] Blockchained log format

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 8 Oct 2017 18:33:35 -0400

The container format is described here. It is also submitted as an ID if
you like unreadable document formats.

http://prismproof.org/Documents/draft-hallambaker-jbcd-container.html

The frame format is binary but the metadata descriptions of the frames can
be text or binary. JSON-B just uses the upper 128 codes in a byte that are
unused in JSON tagging to define a set of tags for binary encoding. So you
have a choice of text or binary for encoding. Since JSON-B is a strict
superset of JSON, one decoder works for both.

The advantage of the double frame pointers is that you can read files just
as fast in either direction. So you can append an index to the end of a
container when it is complete. Or you can drop out incrementals every n
entries. This avoids the need for a second file to maintain an index.

Each frame payload is encrypted separately but a frame can reference the
key exchange parameters in a prior frame. So if you have an online chat,
you might have two different session keys in use at the same time on
interleaved frames.

The idea is to provide for a range of time/memory access tradeoffs. I get
really annoyed when programs take a long time to load because they are
reading an index into memory. When files get large, that becomes
impossible. This approach allows a server to bind to a data repository and
get a 'fast open' with immediate albeit somewhat slower access to any frame
in the file. Meanwhile, an indexer thread can compile a fast access index
if needed.

One tweak I thought about today but have not had time to implement is
external payloads. If a file is really large, e.g. a video file and likely
to be included in more than one log, it might well be desirable to store it
as an external attachment. This might still be wrapped by a frame header
with encryption information.

Yes, cross linking trees/chains is planned but I have not yet thought
through how to do it. I am pretty sure it is a layered feature.

Simplest way at this stage is probably to simply define a content type for
'external digest' and log it as any other payload.

On Thu, Sep 14, 2017 at 10:34 AM, iang <[email-masked]> wrote:

> On 11/09/2017 15:01, Phillip Hallam-Baker wrote:
>
> Ideas? Comments?
>>
>
> First comment - I (also would) do it in binary.  I used to use text
> formats because I could read and confirm the code, but in the end, it was
> easier to be confident of a binary I/O code than text code. And when I
> added encryption, it made the text a bit useless.
>
> Second comment - for better generality, make it two files not one, one
> being indexes and one being the frames in full.  Each index locates its
> frame in the frame file by start / length or end.
>
> This way, you can read the entire index file into memory qiuckly, and then
> only pull in the frames on demand.
>
> I must admit - I never thought about doing the reverse reading. Probably
> because I've always read the whole (index) thing into memory for any and
> all processing.
>
> iang
>
> ps; in my work, both files have the same header, more or less, and the
> header has the encryption keys in it.  A stream cipher is needed, as we're
> picking up at any point in the frame, I use Chacha and poly.
>