libGringotts manual

As we said, libGringotts provides a simple way to process your data, or save them to some file, in an higly configurable and completely secure way, using strong encryption and hashing algorithms to ensure no one will ever be able to read them (well... as long as they don't know the password... ;-). It includes some other useful features, like generation of encrypted temporary files, secure memory management, and functions for the evaluation of password quality.

Here is a complete documentation on libGringotts' interface and internals. First, you'll be presented with the various non-standard data structures and codes you'll use; after these, the functions will be introduced. Maybe you'll want to step right to the examples, which are a sort of tutorial; finally, the very file format will be described.

Table of contents

• Typedefs & enumerations
• Enumerations
• Encapsulations ("objects")
• Error codes
• Functions
• General purpose functions
• libGringotts context-related functions
• libGringotts keyholder-related functions
• Memory-to-memory encryption/decryption functions
• File encryption/decryption functions
• Encrypted Temporary File-related functions
• Miscellaneous utility functions
• An example is better than 10³ words
• In depth: the file formats
• libGringotts file format, version 3
• libGringotts file format, version 2
• libGringotts file format, version 1

In the various functions, you'll use the following types. Some are enumerations, that can take fixed values (also described below), some other just encapsulations, that keep a bunch of data about the context, or other info.

The default values are shown in bold text.

Error codes

When the value returned by a encryption/decryption function is negative, it represents an error. The following values may be used to compare the code with, and find its cause.

The following can be returned by the fuction grg_file_shred():

The following are generic errors, that can be returned by all functions that return an int:

So, here we are. At the very core :-)

unsigned char *grg_get_version (void);

This returns the version of the library, as a string. For example, "1.0.0" or "5.2.22". It's allocated, so you must free() it after use.

unsigned int grg_get_int_version (void);

The version, again, but as a number; this way, you can make comparisons between versions. For example, "1.0.0"=10000 and "5.2.22"=50222, so it's TRUE that 50222 > 10000.

libGringotts context (GRG_CTX) related functions

GRG_CTX grg_context_initialize (const unsigned char *header, const grg_crypt_algo crypt_algo, const grg_hash_algo hash_algo, const grg_comp_algo comp_algo, const grg_comp_ratio comp_lvl, const grg_security_lvl sec_lvl);

Initializes a new libGringotts session, given its parameters. The header is a 3-letter string that specifies which will be the first three bytes of your files, to use as identifier. The other parameters are the algorithms and settings you want to use, as described in the "Enumerations" section, above. If the function returns NULL, an error occoured.

GRG_CTX grg_context_initialize_defaults (const unsigned char *header);

Initializes a new libGringotts session, with some default values. The header is the same as the previous function, and like that one, if the return value is NULL, it indicates an error.

void grg_context_free (GRG_CTX gctx);

Frees the resources encapsulated in a context gctx. It MUST be always called, upon a session termination.

grg_crypt_algo grg_ctx_get_crypt_algo (const GRG_CTX gctx);
grg_hash_algo grg_ctx_get_hash_algo (const GRG_CTX gctx);
grg_comp_algo grg_ctx_get_comp_algo (const GRG_CTX gctx);
grg_comp_ratio grg_ctx_get_comp_ratio (const GRG_CTX gctx);
grg_security_lvl grg_ctx_get_security_lvl (const GRG_CTX gctx);

Gets the various settings encapsulated in a context.

void grg_ctx_set_crypt_algo (GRG_CTX gctx, const grg_crypt_algo crypt_algo);
void grg_ctx_set_hash_algo (GRG_CTX gctx, const grg_hash_algo hash_algo);
void grg_ctx_set_comp_algo (GRG_CTX gctx, const grg_comp_algo comp_algo);
void grg_ctx_set_comp_ratio (GRG_CTX gctx, const grg_comp_ratio comp_ratio);
void grg_ctx_set_security_lvl (GRG_CTX gctx, const grg_security_lvl sec_level);

These functions changes the settings for a given context, to adapt its future behaviour to the programmer's needings.

unsigned int grg_get_key_size_static (const grg_crypt_algo crypt_algo);
unsigned int grg_get_key_size (const GRG_CTX gctx);
unsigned int grg_get_block_size_static (const grg_crypt_algo crypt_algo);
unsigned int grg_get_block_size (const GRG_CTX gctx);

Intuitively enough, they obtain informations on the size of block and key of an ancryption algorithm; they can be obtained by the algorithm id (the _static version of each) or by the one encapsulated in a specific context.

libGringotts keyholder (GRG_KEY) related functions

GRG_KEY grg_key_gen (const unsigned char *pwd, const int pwd_len);

Generates a new keyholder, from a password pwd (that can be any byte sequence), of length pwd_len. If pwd_len is -1, the length is autodetected, and the password must be NULL-terminated. It returns NULL on errors.
Note: the password is not stored in plaintext, and the memory occupied by the keyholder is fixed and does not depend on the password size.

GRG_KEY grg_key_clone (const GRG_KEY src);

Clones a keyholder src into a new one.

int grg_key_compare (const GRG_KEY k1, const GRG_KEY k2);

Compares two keyholders, k1 and k2. If they are equals, it returns 0; else, 1.

void grg_key_free (const GRG_CTX gctx, GRG_KEY key);

Releases the resources associated with the keyholder key, using the context gctx. It securely wipes it, so no one can recover its data later on.

int grg_validate_mem (const GRG_CTX gctx, const void *mem, const long memDim);

Validates a memory segment containing libGringotts' data. In other words, it simply says if it's valid or not, and if not, why. Returns one of the error codes described above. The memory address is located at mem, of length memDim.

int grg_update_gctx_from_mem (GRG_CTX gctx, const void *mem, const long memDim);

Detects the algorithms used to produce a libGringotts-encoded memory segment (at mem, of length memDim), and updates the context gctx to contain these data.

int grg_encrypt_mem (const GRG_CTX gctx, const GRG_KEY keystruct, void **mem, long *memDim, const long origDim);

Encodes data to a memory region. The encryption is performed with the password contained in keystruct and the settings specified in gctx. The encoded data produced will be written to a newly-allocated (to free after use) memory region, pointed by *mem, and its length will be written to *memDim. origData and origDim are, respectively, the data to be encoded (which can be any byte sequence, not just strings) and their length. As ever, if origDim = -1 the length is autodetected, and origData must be NULL-terminated. The return value is a libGringotts error code, to compare in the table above.

int grg_decrypt_mem (const GRG_CTX gctx, const GRG_KEY keystruct, const void *mem, const long memDim, unsigned char **origData, long *origDim);

Reads data from a libGringotts-encoded data sequence, located at mem (of size memDim), using the password in keystruct. The context gctx is adopted, and updated with the algorithms used to encrypt the file. It returns an error code in case of errors. The read data are stored in origData, and their length in origLen, that can be NULL if you don't want to retrieve length. The former is allocated dinamically, so you'll want to free() (better, grg_free() ;-) it after use.

These are basically the same functions as above, but they read and save data from files instead than memory. If you work with files, please use these, because they are more integrated than operating with memory and then interface it on files, resulting in faster & safer I/O.

int grg_validate_file (const GRG_CTX gctx, const unsigned char *path);

Validates a libGringotts' file (at path). In other words, it simply says if it's valid or not, and if not, why. Returns one of the error codes described above.

int grg_update_gctx_from_file (GRG_CTX gctx, const unsigned char *path);

Detects the algorithms used to produce a file (at path), and updates the context gctx to contain these data.

int grg_encrypt_file (const GRG_CTX gctx, const GRG_KEY keystruct, const unsigned char *path, const unsigned char *origData, const long origDim);

Writes data to a new libGringotts file. The encryption is performed with the password contained in keystruct and the settings specified in gctx; the file produced will be written to path. origData and origDim are, respectively, the data to be encoded (which can be any byte sequence, not just strings) and their length. As ever, if origDim = -1 the length is autodetected, and origData must be NULL-terminated. The return value is a libGringotts error code, to compare in the table above.

int grg_decrypt_file (const GRG_CTX gctx, const GRG_KEY keystruct, const unsigned char *path, unsigned char **origData, long *origDim);

Reads data from an encrypted file located at path, using the password in keystruct. The context gctx is adopted, and updated with the algorithms used to encrypt the file. It returns an error code in case of errors. The read data are stored in origData, and their length in origLen, that can be NULL if you don't want to retrieve length. The former is allocated dinamically, so you'll want to free() (better, grg_free() ;-) it after use.

There is also a "direct" version of each of these functions, that accepts an already opened file descriptor instead of a filename. This may be desirable to avoid race conditions, i.e. when validating a file before actually opening it. Notice that these don't close the file descriptor; that operation is up to you.

int grg_validate_file_direct (const GRG_CTX gctx, const int fd);
int grg_update_gctx_from_file_direct (GRG_CTX gctx, const int fd);
int grg_encrypt_file_direct (const GRG_CTX gctx, const GRG_KEY keystruct, const int fd, const unsigned char *origData, const long origDim);
int grg_decrypt_file_direct (const GRG_CTX gctx, const GRG_KEY keystruct, const int fd, unsigned char **origData, long *origDim);

GRG_TMPFILE grg_tmpfile_gen (const GRG_CTX gctx);

Creates a new encrypted tempfile, and returns an handler to it. The file will be written with the encryption algorithm contained in the context at this time; any modification to it will not be reflected on the file. The file is encrypted with a random key, unavailable to the user. The handler will be used to write (once) data to the file, and retrieve them later.

int grg_tmpfile_write (const GRG_CTX gctx, GRG_TMPFILE tf, const unsigned char *data, long data_len);

This function is used to write data (of length data_len, -1 as ever if data is NULL-terminated) in the temporary file represented by tf. This can be done only once. The file, after this, becomes read-only. Returns an error code as usual.

int grg_tmpfile_read (const GRG_CTX gctx, GRG_TMPFILE tf, unsigned char **data, long *data_len);

This function is used to read data from a temporary file. The length is stored in data_len, but you can specify NULL if you don't want to retrieve it. This operation can be done only after a successful writing. Returns an error code as usual.

void grg_tmpfile_close (const GRG_CTX gctx, GRG_TMPFILE tf);

Releases all the resources incapsulated in the tmpfile handler, in a secure way.

unsigned char *grg_rnd_seq (const GRG_CTX gctx, const unsigned int size);
void grg_rnd_seq_direct (const GRG_CTX gctx, unsigned char *toOverwrite, const unsigned int size);

Returns a random byte sequence, basing on the context gctx, of size size. The _direct version overwrites a (valid) string provided by the user (it's his responsibility to check that string and size are coherent; if size = -1 it will be autodetected), the other allocates a new string, to free() afterwards.

unsigned char grg_rnd_chr (const GRG_CTX gctx);

Returns a random byte.

void grg_free (const GRG_CTX gctx, void *alloc_data, const long dim);

Frees allocated memory (pointed by alloc_data), overwriting it with random trash. Precious when you have sensitive data in memory, and you do want to ensure that they aren't readable in any way, afterwards. dim is the dimension of the data to wipe, or -1 if it's NULL-terminated.

double grg_ascii_pwd_quality (const unsigned char *pwd, const long pwd_len);

Gives an estimation of the quality of a string password, on a scale from 0 to 1. pwd_len is the length of the password pwd; you can specify -1 if it's NULL-terminated.

double grg_file_pwd_quality (const unsigned char *pwd_path);

Gives an estimation of the quality of a file (on the disk, at pwd_path), when using it as a password, on a scale from 0 to 1.

unsigned char* grg_encode64 (const unsigned char *in, const unsigned int inlen, unsigned int *outlen);
unsigned char* grg_decode64 (const unsigned char *in, const unsigned int inlen, unsigned int *outlen);


These function convert a generic sequence of bytes to/from the base64 format. Useful for email sending, xml storing, etc.

int grg_file_shred (const char *path, const int npasses);

Securely wipes a file, overwriting it npasses times with random data. The data can't be recovered, once done this; be careful. This option still have some limitations, as it takes the assumption that the filesystem overwrites files in place: for some FSs this isn't true. See man 1 shred for details.

Confused? Don't be! It's simple! :-) Here is a couple of examples of how to tie it all together. This is a piece of code that writes data to a file:

#include <libgringotts.h>
#include <string.h>

[...]

{
	//initialization of structures
	GRG_CTX context = grg_context_initialize_defaults ("TRY");
	GRG_KEY keyholder = grg_key_gen ("This is a nice password", -1);

	//let's produce some data to encode
	unsigned char *plaintext = "These are some valuable and important data!";
	unsigned int txtlen = strlen (plaintext);
	unsigned char *filepath = "/home/mano/somefile.try"; //extension is not relevant

	//now we change some settings, and leave the others as defaults
	grg_ctx_set_crypt_algo (context, GRG_TWOFISH);
	grg_ctx_set_comp_algo (context, GRG_BZIP);

	//finally, let's save!
	grg_encrypt_file (context, keyholder, filepath, plaintext, txtlen); //txtlen could be -1

	//always remember to clean up leftovers...
	grg_key_free (context, keyholder);
	grg_context_free (context);
}

[...]

To decode, things are even simpler:

#include <libgringotts.h>
#include <stdio.h>

[...]

{
	//again, initialization of structures
	//or you can reuse the same as before
	GRG_CTX context = grg_context_initialize_defaults ("TRY");
	GRG_KEY keyholder = grg_key_gen ("This is a nice password", -1);

	unsigned int *filepath = "/home/mano/somefile.try";

	//here the library will put the decoded data
	unsigned char *plaintext;
	long int txtlen;

	//we are ready to perform the decryption
	grg_decrypt_file (context, keyholder, filepath, &plaintext, &txtlen);

	//now, data are stored in the two variables and in the
	//context, that's changed accordingly. Let's use them...
	printf ("Decrypted data are %d long, and are \"%s\"\n", txtlen, plaintext);
	if (grg_ctx_get_crypt_algo (context) == GRG_TWOFISH)
		printf ("You encrypted the file with Twofish\n");
	else
		printf ("You didn't encrypt the file with Twofish\n");

	//as always, remember to clean up leftovers...
	grg_key_free (context, keyholder);
	grg_free (context, plaintext, txtlen);
	grg_context_free (context);
}

[...]

This silly thing writes some stuff to a temporary file, then reads it.

[...]

{
	//init of structs
	GRG_CTX gctx = grg_context_initialize_defaults ("tRy");
	GRG_TMPFILE tf = grg_tmpfile_gen (gctx);
	unsigned char *try;
	long int dim;

	//writes some stupid data
	grg_tmpfile_write (gctx, tf, "S0M3 S7UP1D DA7A", 16);

	[...]

	//then reads it. Simple...
	grg_tmpfile_read (gctx, tf, &try, &dim);

	//Freedom... freedom... freedom... ;-)
	grg_tmpfile_close (gctx, tf);
	grg_context_free (gctx);

	//Use those data!
	printf("Data were %s, of length %ld\n", try, dim);
}

[...]

A description of the inners of the libGringotts File Format follows. Please use it... in any way you like! ;-)

A file has this structure:

HEADER | VERSION | CRC32 | ALGO | IV | CRC32 | DATA_LEN | DATA

• the part written in italic is encrypted;
• the part written in bold is compressed and encrypted

• HEADER: an unique file format ID [it must be 3 bytes long]
• VERSION: the file format version ("3") [1b]
• CRC32: the CRC32 of the remaining file part [4b]
  The second is used to check the correctness of the password
• ALGO is a bitfield defined as follow:

     IAAABCDD
  MSB^      ^LSB
  

  where:
  • I: ignored, set it to 0
  • A: encryption algorithm
  • 000 -> AES (Rjindael-128)
  • 001 -> Serpent
  • 010 -> Twofish
  • 011 -> CAST-256
  • 100 -> SAFER+
  • 101 -> LOKI97
  • 110 -> 3DES
  • 111 -> Rjindael-256
  • B: hashing algorithm
  • 0 -> SHA-1
  • 1 -> RIPEMD-160
  • C: compression type
  • 0 -> ZLib
  • 1 -> libBZ2
  • D: compression level
  • 00 -> none (0)
  • 01 -> fast (3)
  • 10 -> good (6)
  • 11 -> best (9)
  in bold the defaults (0 001 1 0 11 = 0x1b).
• IV is the Initialization Vector of the encryption function. It's randomly generated at file encryption, and its dimension varies from 8 to 32 bytes accordingly to encryption algorithm block size
• DATA_LEN: the length of the uncompressed DATA [4b]
• DATA: the data stored in this file

This way, the final file length is typically (compressed DATA)+33b, and this format allows compressed data size up to 512 Mb ( 2^4*8 ). Fair enough.

The data are encrypted using the given algorithm with a key derived from the password. This key is calculated hashing the password data with the chosen algorithm, for the length required by the encryption algorithm, and using the S2K algorithm KEYGEN_S2K_SIMPLE of MHash. The key obtained is XOR'ed with the IV, repeated as many times as needed; this way, I have a different key for each saving of the file, increasing security.

This was analogous to format 3, but used KEYGEN_MCRYPT for key generation; Bob Mathews found that this had a minor weakness, so I changed it. It isn't supported anymore from libGringotts 1.0.0; use Gringotts 0.7.0 to convert an old file in this format.

It didn't allow any choice of the algorithms; it used SHA-256 for hashing, and SERPENT for encryption.

It isn't supported anymore, starting from libGringotts 2 (Gringotts 0.6.0). If you still have some file encrypted with this, use an older Gringotts (between 0.4.2 and 0.5.9) to convert it to v2, and then Gringotts 0.7.0 to convert to v3.