Jonas Plum

Apple container

Sun, 28 Jun 2026 00:00:00 +0000

I tried https://github.com/apple/container today. It's a pretty nice docker replacement and works well on macOS 15.5, even though officially only macOS 26 is supported. The only thing I'm missing currently is file mounts.

Use markdown as config

Sun, 26 May 2024 00:00:00 +0000

Most of the time we use json or yaml files for configuration. But these lack the ability to display styled text or syntax-highlighted code samples. Markdown has these capabilities, but cannot be easily parsed to be used as a configuration file.

I created https://github.com/cugu/md to parse markdown into Go structs, similar to how encoding/json.Unmarshal works for json files.

The following example shows how to use the package.

 1package main
 2
 3import (
 4	"fmt"
 5
 6	"github.com/cugu/md"
 7)
 8
 9type Text struct {
10	Title       string `md:"heading"`
11	Description string `md:"paragraph"`
12}
13
14const example = `
15# Title
16A short description.
17`
18
19func main() {
20	var text Text
21	if err := md.Unmarshal([]byte(example), &text); err != nil {
22		fmt.Println(err)
23		return
24	}
25
26	fmt.Println(text.Title)
27	fmt.Println(text.Description)
28	// Output:
29	// Title
30	// A short description.
31}

Use case 1: Package definitions #

I started to create a simple package manager which uses markdown files for package definitions. Sections are used for the install and uninstall commands. GitHub renders these files as markdown, so headings and description paragraphs render nicely for humans, and the embedded code is syntax-highlighted as well.

 1# Starship prompt
 2
 3Starship is a minimal, blazing-fast, and infinitely customizable prompt for any shell!
 4
 5## Install
 6
 7```bash
 8#!/bin/bash
 9
10# download starship and install it
11curl -fsSL https://starship.rs/install.sh -o install-starship.sh
12chmod +x install-starship.sh
13./install-starship.sh --bin-dir ~/bin --yes
14
15# more steps...
16```
17
18## Uninstall
19
20```bash
21...

Use case 2: End-to-end tests #

Another project I am working on processes received json events. This project uses markdown files for end-to-end testing. The input json and the expected output logs are defined in different markdown sections. Titles and paragraphs are used for description.

 1# Test 1: Reply to Slack message
 2
 3Reply to a received Slack [message event](https://api.slack.com/events/message).
 4
 5## Event
 6
 7```json
 8{
 9  "type": "message",
10  "channel": "C123ABC456",
11  "user": "U123ABC456",
12  "text": "Hello Bob",
13  "ts": "1355517523.000005"
14}
15```
16
17## Output
18
19```log
20INFO "received message" type="message" text="Hello Bob"
21INFO "reply sent" msg="Hello Alice"
22```

Conclusion #

I'm happy if https://github.com/cugu/md is useful for your projects and for your feedback. I also consider adding features, e.g. marshalling if needed, but want to keep the package as simple as possible.

A take on supply chain security in Go

Wed, 12 Jan 2022 00:00:00 +0000

List your dependencies capabilities and monitor if dependency updates require more capabilities.

The Problem #

Recently different attacks and other issues related to open-source dependencies highlighted a quite severe problem with dependencies:

Every imported package gives that package's author basically remote code execution for your software.

The Idea #

A video on WASI by linclark brought me to the idea how cool it would be if we could pass permissions to our dependencies.

In Go this could look something like this:

module github.com/cugu/gocap

go 1.20

require (
	github.com/go-chi/chi       v5.0.7   (network:read)
	github.com/mattn/go-sqlite3 v1.14.10 (file:read, file:write)
	github.com/sirupsen/logrus  v1.8.1   (os:stdout)
	github.com/yuin/goldmark    v1.4.4   ()
)

chi would be able to receive network requests, go-sqlite3 would be able to read and write files and logrus could write to stdout. But also all those modules would be limited to those capabilities and, for example, the logging library logrus would not be able to interact with files, the network or execute code.

Changes of dependencies would be much less critical in many cases, as a potential attacker would have only limited attack surface besides stealing your CPU cycles.

A simpler but working approach: GoCap #

Implementing the approach above would require changes to Go itself. So I came up with another, simpler approach: GoCap. GoCap can check and validate the source code of dependencies for their capabilities and is ment to be included into the testing phase of the build process. This way GoCap can at least pin the capabilities of dependencies.

GoCap provides simple capability checking for Go using a go.cap file. The go.cap files lists all package dependencies that require critical permissions like file access, execution rights or network access. Unlike the idea above GoCap works on packages not modules and capabilities are based on the imported packages of the standard library.

The go.cap file for GoCap itself looks like this:

github.com/cugu/gocap (execute, file)

github.com/alecthomas/kong (file, syscall)
github.com/pkg/errors (runtime)

Install GoCap #

You can download a release or run

1go install github.com/cugu/gocap@v0.1.0

gocap generate #

gocap generate <path> prints a valid go.cap file. It lists all dependency packages that require critical permissions like file access, execution rights or network access.

! gocap generate runs on package basis and the path argument must point to a Go package (there must be .go files) not a Go Module. Also gocap generate needs the dependencies downloaded before, e.g. run go mod download.

Example

1cd ~/myProjects/gocap
2go mod download
3gocap generate . > go.cap

go.cap

github.com/cugu/gocap (execute, file)

github.com/alecthomas/kong (file, syscall)
github.com/pkg/errors (runtime)

gocap check #

gocap check <path> compares a local go.cap file with the actual required capabilities by dependency packages. Any mismatch results in a non-zero exit code, so you can use GoCap check in your CI pipelines. See ci.yml for a working example.

Example

1gocap check .

Output

github.com/alecthomas/kong
	capability 'syscall' not provided by go.cap file, add to go.cap file if you want to grant the capability
github.com/pkg/errors
	unnecessary capability 'network', please remove from go.cap file

Capabilities #

Name	Description	Packages
file	Read and write access to files	os io/ioutil
network	Read and write to the network	net net/http
execute	Execute other binaries	os/exec
syscall	Perform any system call in context of the software	syscall, C
unsafe	Usage of the unsafe package in combination with a //go:linkname compiler directive can be used to load external C or assembler code	unsafe
reflect	reflect.NewAt in combination with reflect#Value.Call might be used call arbitrary functions	reflect

APFS filesystem format

Sat, 22 Apr 2017 00:00:00 +0000

I started to reverse engineer APFS and want to share what I found out so far. You can send me feedback and ideas on this post via Twitter.

Notice: I created a test image with macOS Sierra 10.12.3 (16D32). All results are guesses and the reverse engineering is work in progress. Also newer versions of APFS might change structures. The information below is neither complete nor proven to be correct.

Update 2017-04-30: Added a section for the checksum Update 2017-06-16: Add apfs.ksy respository

Contents #

Overview #

APFS is structured in a single container that can contain multiple APFS volumes. A container needs to be >512 MB to contain more than one volume, >1024MB to contain more than two volumes and so on. The following image shows an overview of the APFs structure.

Each element of this structure (except for the allocation file) starts with a 32 byte block header, which contains some general information about the block. Afterwards the body of the structure is following. The following types exist:

0x01: Container Superblock
0x02: Node
0x05: Spacemanager
0x07: Allocation Info File
0x11: Unknown
0x0B: B-Tree
0x0C: Checkpoint
0x0D: Volume Superblock

Each of this structures is described in detail below. A more detailed version of the APFS structure is available as a Kaitai struct file: apfs.ksy. You can use it to examine APFS dumps in the Kaitai IDE or create parsers for various languages. This .ksy file must considered experimental.

General information: #

The filesystem uses litte-endian values for storing information
Timestamps are 64bit nanoseconds (1 / 1,000,000,000 seconds!) starting from 1.1.1970 UTC (unix epoch). The current timestamp is around 0x14b11800f375e000.
Standard block size seems to be 4096 byte per block.
APFS is a copy-on-write filesystem so each block is copied before changes are applied so a history of all unoverwritten files and filesystem structures exists. This might result in a huge amount of forensic artefacts.

Structures #

Block header #

Each filesystem structure in APFS starts with a block header. This header starts with a checksum for the whole block. Other informations in the header include the copy-on-write version of the block, the block id and the block type.

pos	size	type	id
0	8	uint64	checksum
8	8	uint64	block_id
16	8	uint64	version
24	2	uint16	block_type
26	2	uint16	flags
28	4	uint32	padding

Checksum #

According to the apple docs the Fletcher's checksum algorithm is used. Apple uses a variant of the algorithm described in a paper by John Kodis. The following algorithm shows this procedure. The input is the block without the first 8 byte.

 1func createChecksum(data []byte) uint64 {
 2    var sum1, sum2 uint64
 3
 4    modValue := uint64(2<<31 - 1)
 5
 6    for i := 0; i < len(data)/4; i++ {
 7        d := binary.LittleEndian.Uint32(data[i*4 : (i+1)*4])
 8        sum1 = (sum1 + uint64(d)) % modValue
 9        sum2 = (sum2 + sum1) % modValue
10    }
11
12    check1 := modValue - ((sum1 + sum2) % modValue)
13    check2 := modValue - ((sum1 + check1) % modValue)
14
15    return (check2 << 32) | check1
16}

The nice feature of the algorithm is, that when you check a block in APFS with the following algorithm you should get null as a result. Note that the input in this case is the whole block, including the checksum.

 1func checkChecksum(data []byte) uint64 {
 2    var sum1, sum2 uint64
 3
 4    modValue := uint64(2<<31 - 1)
 5
 6    for i := 0; i < len(data)/4; i++ {
 7        d := binary.LittleEndian.Uint32(data[i*4 : (i+1)*4])
 8        sum1 = (sum1 + uint64(d)) % modValue
 9        sum2 = (sum2 + sum1) % modValue
10    }
11
12    return (sum2 << 32) | sum1
13}

Container Superblock #

The container superblock is the entry point to the filesystem. Because of the structure with containers and flexible volumes, allocation needs to handled on a container level. The container superblock contains information on the blocksize, the number of blocks and pointers to the spacemanager for this task. Additionally the block IDs of all volumes are stored in the superblock. To map block IDs to block offsets a pointer to a block map b-tree is stored. This b-tree contains entries for each volume with its ID and offset.

pos	size	type	id
0	4	byte	magic "NXSB"
4	4	uint32	blocksize
8	8	uint64	totalblocks
40	16	byte	guid
56	8	uint64	next_free_block_id
64	8	uint64	next_version
104	4	uint32	previous_containersuperblock_block
120	8	uint64	spaceman_id
128	8	uint64	block_map_block
136	8	uint64	unknown_id
144	4	uint32	padding2
148	4	uint32	apfs_count
152	8	uint64	offset_apfs (repeat apfs_count times)

Node #

Nodes are flexible containers that are used for storing different kinds entries. They can be part of a B-tree or exist on their own. Nodes can either contain flexible or fixed sized entries. A node starts with a list of pointers to the entry keys and entry records. This way for each entry the node contains an entry header at the beginning of the node, an entry key in the middle of the node and an entry record at the end of the node.

pos	size	type	id
0	4	uint32	alignment
4	4	uint32	entry_count
10	2	uint16	head_size
16	8	entry	meta_entry
24	...	entry	entries (repeat entry_count times)

Spacemanager #

The spacemanager (sometimes called spaceman) is used to manage allocated blocks in the APFS container. The number of free blocks and a pointer to the allocation info file(s?) are stored here.

pos	size	type	id
0	4	uint32	blocksize
16	8	uint64	totalblocks
40	8	uint64	freeblocks
144	8	uint64	prev_allocationinfofile_block
352	8	uint64	allocationinfofile_block

Allocation Info File #

The allocation info file works as a missing header for the allocation file. The allocation files length, version and the offset of the allocation file are stored here.

pos	size	type	id
4	4	uint32	alloc_file_length
8	4	uint32	alloc_file_version
24	4	uint32	total_blocks
28	4	uint32	free_blocks
32	4	uint32	allocationfile_block

Unknown #

The structure with type 0x11 is quite empty and seems to be related to the spacemanager as it occurs adjacent to it. Its purpose it unknown.

B-Tree #

B-trees manage multiple nodes. They contain the offset of the root node.

pos	size	type	id
16	8	uint64	root

Checkpoint #

A checkpoint structure exists for every container superblock. But I have no clue what it is good for.

Volume Superblock #

A volume superblock exists for each volume in the filesystem. It contains the name of the volume, an ID and a timestamp. Similarly to the container superblock it contains a pointer to a block map which maps block IDs to bock offsets. Additionally a pointer to the root directory, which is stored as a node, is stored in the volume superblock.

pos	size	type	id
0	4	byte	magic "APSB"
96	8	uint64	block_map
104	8	uint64	root_dir_id
112	8	uint64	pointer3
120	8	uint64	pointer4
208	16	byte	guid
224	8	uint64	time1
272	8	uint64	time2
672	8	str(ASCII)	name

Allocation File #

Allocation files are simple bitmaps. They do not have a block header and therefore no type id.

Structure of gpg keys

Tue, 24 Jun 2014 00:00:00 +0000

Creating a new nice pgp key I first needed to examine the exact structure of a pgp key.

I used the batch mode of gpg (gpg --status-fd 1 --no-tty --batch --gen-key) to create the key 0x4B9E3333 with the settings:

Key-Type: default
Subkey-Type: default
Name-Real: Cugu
Name-Email: cugu@cugu.eu
Expire-Date: 2020-05-04
Passphrase: password
%pubring foo.pub
%secring foo.sec
%commit

A nice feature here, is that the short name Cugu is allowed, while in command line mode at least five letters are mandatory.

The created public gpg key consists of five single packets.

Public Key Packet #

The first packet contains the public key information as well as creation date. The created key is an RSA key so the public key is the e which is 0x10001 (=65537) in this case. The 2048 bit (256 byte) long n is the modulus for the RSA operation.

The owner of the secret key has the secret RSA exponent d. To sign a message the owner can calculate m^d mod n = c. Using this public key everyone can afterwards calculate c^e mod n = m₁ and validate that m₁ = m.

99
=======
  01 0D       269 length
  04          Version 4
  53946f6e    Creation Date Timestamp
  01          RSA
  0800        2048
  bfde487046d4d5b5460a4682a21ea8b874675a67ffe5a3a9685d45f02a8ad420
  b08389f200c6e3cae3b113ef81150ef4bacd15f7c496ee5aca41b18d440d8449
  bb791131b3c25ee54c9b10394b4740d70402a9d9316f5ddb9aa741b512b81099
  837b46a7f04d11feaa5b69ed630e8c7a94f6c7754f62634ec1663b00aac3c957
  23798c69abb619be691e0c990d93e83da2c4d2269ebe1fdece5a517c83c0b567
  1e9c4da6010d3b74baada0289b64f24bd6398a6ae652d4b4745ab908955ff04d
  a3f87c5ee087bacef35f6f4507f9becdb16425e4d94ae633168ace98e0eb3110
  ebd9ce358bc07eb20f411b52cb4334cfdd46ee6d6da8f0f10f83e4ff329a4a41
              RSA n
  0011        Padding?
  010001      RSA e

User ID Packet #

This second packet contains the data that should be bound to the public key from above.

b4
=======
  13          Length 19
  43756775203c6375677540637567752e65 Cugu <cugu@cugu.eu>
  753e

Signature Packet #

This packet binds the User ID Packet to the public key. The content of all the subpackets can be gathered by using pgpdump or reading RFC4880.

The key flags sub packet indicates, that this key can just be used for signing and certification. openpgp shows sc in this case. A capital letter shows, that a sub key is capable of other functions (e.g. capital E -> sub key can be used for encryption)

The most interesting part here is sig(i). i is build by a combination of a lot of information:

The complete Public Key Packet (e.g. 0x99010D...010001)
The user ID Packet, where the length is encoded in 4 byte. (e.g. in our case: 0xb40000001343756775203c6375677540637567752e65753e)
Data from this Signature Packet starting including version till the end of the hashed subpackets (e.g. 0x0413010a...021780)
A trailer starting with 0x04FF and a 4 byte length of the data from the Signature Packet

All this information is concatenated and hashed. This hash is afterwards appended to a combination of 0x1f followed by many 0xff a single 0x00 and a specification of the used hash (e.g. i=0x1ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff003051300d060960864801650304020305000440 + hash)

The value 0x3051300d060960864801650304020305000440 for our hash is specified in RFC4880 5.2.2.

All this values (i=0x1f...) is signed: i^d mod n = sig(i)

89
=======
  013d        Length 317
  04          Version
  13          Positive certification of a User ID and Public-Key packet.
  01          RSA
  0a          SHA512

  00 27         Length hashed Subpackets 15

    05          Length 5
    02          Subpacket Type (2 = Signature Creation Time)
    53946f6e    Timestamp

    02          Length 2
    1b          Subpacket Type (27 = Key Flags)
    03

    05          Length 5
    09          Subpacket (Type 9 = Key Expiration Time)
    0b1b7832    Timestamp Offset (186349618 + Creation Time)

    05          Length 5
    0b          Subpacket (Type 11 = Preferred Symmetric Algorithms)
    09080703

    05          Length 5
    15          Subpacket (Type 21 = Preferred Hash Algorithms)
    0a09080b

    05          Length 5
    16          Subpacket (Type 22 = Preferred Compression Algorithms)
    02030100

    02          Length 2
    1e          Subpacket (Type 30 = Features)
    01          Modification Detection

    02          Length 2
    17          Subpacket (Type 23 = Key Server Preferences)
    80          0x80 = No-modify

  00 0a         Length unhased Subpackets 10

    09          Length 9
    10          Subpacket (Type 16 = Issuer)
    48c1 f448 4b9e 3333
                last 4 blocks of the fingerprint

  c678        last 2 byte of the hash over m

  0800        2048
  816a53faf1135895ad8890c165a32a1d1166e5df41fa43713f02fafdbc55f4f1
  848038f55f778550acb76ebab3d6323604342445f7224406edab06c93f46c246
  b109b4b18a1ccc71fda28e3f1059d866c2e389f9ab589e76be11e5ef737e1178
  f5675425d5f5c4246a04e76d5dc9dc6b93550f0766a57298ed463a9fc6e8bba8
  3287f3b5c33eebefbf358e32936e0a316be86d56ac96391e796b8eac7091e215
  6a7eae9082369775901c86fcb1ca6c74d8c345d9db6c299923a86d97287209f9
  9bedf83b599e561fff370877dd1a7c6fecd632b2d984e7b1c27fbfc80e3a59e2
  217b0708b161192e194c204506aed50b20485da514dc29df2128f8ac42fe2458
              sig(i)

Public Subkey Packet #

This subkey can be used for encryption. Otherwise this and the following signature packet are built the same way as above.

b9
=======
  010d        269 length
  04          Version 4
  53946f6e    Timestamp
  01          RSA
  0800        2048
  aa19801ed96301caaef25633b87eedcecfb31a7363fd798264b1f9fa307728ed
  b50f086404930ecbf03f13d807e181ade29268589eba63c459b6e90388d40169
  29a63d6e06c4f87289aef3269cd5ba9d31bce6f364ea97513611a84e97a85f32
  6b77114e0c4ec3c5249f91fd48783713d0825ef99db56e846562189844454722
  67cd4e64d43fa2a305267c0482b968ec00e8cea00b6464ef6e4117095036779a
  4b713ec1ad04580b4fcebb66ed8afc3a0e6c315f6790a0c6f2277ae8a017fe9e
  67b036cc786d85718cc1d0670dc1c7d688369eb10934e1b3245a183ab70e3068
  80cb3f91c8fd389c5e7f46bfd1e3fb70def93900bfbe2cb59faaa1df311d253f
              512 bit
  0011        Padding?
  010001      RSA e

Signature Packet #

89
=======
  0125        293 Length
  04          Version 4
  18          Subkey Binding Signature
  01          RSA
  0a          SHA512

  00 0f         Length hashed Subpackets 15

    05          Length 5
    02          Subpacket Type 2 = Signature Creation Time
    53946f6e    Timestamp

    02          Length 2
    1b          Subpacket Type 27 = Key Flags
    0c          1100

    05          Length 5
    09          Subpacket (Type 9 = Key Expiration Time)
    0b1b7832    Timestamp Offset (186349618 + Creation Time)

  00 0a         Length unhased Subpackets 10

    09          Length 9
    10          Subpacket (Type 16 = Issuer)
    48c1f4484b9e3333
                Last 4 Blocks FP

  c16e        diff

  0800        2048
  a9b0401874d80f839c407b50e5dcb5ee51e5aafc95933fee013631620a61e69d
  63be81e79f35b57cb4cf372ea7bdaa23e9593e18a747101229849cd59517bd75
  e8d2965ccf517420906fb649e6f885f927bcabd9f925b80a31d165b12776260f
  711ed37d10262d1c103b42fd0baead95529d6c2f10cae24d353dc1e9aa2336ce
  9457c4d484eed4416ec1a23a734ca41d146fc79229757a5b7a61f2ac5600cd4c
  f800aa118e77dbd32d20c774021dd70f2efb90f00218593e3b00fe97ecbdfc05
  633fee7c75bb27aa44d76d9c8c047e7bb1c1d07212c668639bbab1ed9c1288f9
  b914da5f600baaa22e911338205c58b2bc1d3282ff54e7801eb8889284686eb3

Recover broken docx

Tue, 13 May 2014 00:00:00 +0000

Today a friend of me asked me to recover a docx document. MS Word called this document broken.

docx is essentially a zip file. The command zip -F broken.docx --out restore.zip tries to restore the file. Well no success. zip -FF broken.docx --out restore.zip tries harder! Afterwards unzip restore.zip unzips the file. The file document.xml in the folder word contains the actual text data from the word document. Unfortunately in my case the document.xml was half empty :( Zip commands are taken from: http://computers.tutsplus.com/tutorials/how-to-repair-damaged-zip-files-with-terminal--mac-49294