Base64 Support for output formatting and AVRO Serialisation

apply PR https://github.com/edenhill/kcat/pull/206/files
Also adds a base64 format option for the the payload and partition keys

LICENSE.base64

@@ -0,0 +1,126 @@
+# $FreeBSD$
+#	@(#)COPYRIGHT	8.2 (Berkeley) 3/21/94
+
+The compilation of software known as FreeBSD is distributed under the
+following terms:
+
+Copyright (c) 1992-2019 The FreeBSD Project.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGE.
+
+The 4.4BSD and 4.4BSD-Lite software is distributed under the following
+terms:
+
+All of the documentation and software included in the 4.4BSD and 4.4BSD-Lite
+Releases is copyrighted by The Regents of the University of California.
+
+Copyright 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
+	The Regents of the University of California.  All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+3. All advertising materials mentioning features or use of this software
+   must display the following acknowledgement:
+This product includes software developed by the University of
+California, Berkeley and its contributors.
+4. Neither the name of the University nor the names of its contributors
+   may be used to endorse or promote products derived from this software
+   without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGE.
+
+The Institute of Electrical and Electronics Engineers and the American
+National Standards Committee X3, on Information Processing Systems have
+given us permission to reprint portions of their documentation.
+
+In the following statement, the phrase ``this text'' refers to portions
+of the system documentation.
+
+Portions of this text are reprinted and reproduced in electronic form in
+the second BSD Networking Software Release, from IEEE Std 1003.1-1988, IEEE
+Standard Portable Operating System Interface for Computer Environments
+(POSIX), copyright C 1988 by the Institute of Electrical and Electronics
+Engineers, Inc.  In the event of any discrepancy between these versions
+and the original IEEE Standard, the original IEEE Standard is the referee
+document.
+
+In the following statement, the phrase ``This material'' refers to portions
+of the system documentation.
+
+This material is reproduced with permission from American National
+Standards Committee X3, on Information Processing Systems.  Computer and
+Business Equipment Manufacturers Association (CBEMA), 311 First St., NW,
+Suite 500, Washington, DC 20001-2178.  The developmental work of
+Programming Language C was completed by the X3J11 Technical Committee.
+
+The views and conclusions contained in the software and documentation are
+those of the authors and should not be interpreted as representing official
+policies, either expressed or implied, of the Regents of the University
+of California.
+
+
+NOTE: The copyright of UC Berkeley's Berkeley Software Distribution ("BSD")
+source has been updated.  The copyright addendum may be found at
+ftp://ftp.cs.berkeley.edu/pub/4bsd/README.Impt.License.Change and is
+included below.
+
+July 22, 1999
+
+To All Licensees, Distributors of Any Version of BSD:
+
+As you know, certain of the Berkeley Software Distribution ("BSD") source
+code files require that further distributions of products containing all or
+portions of the software, acknowledge within their advertising materials
+that such products contain software developed by UC Berkeley and its
+contributors.
+
+Specifically, the provision reads:
+
+"     * 3. All advertising materials mentioning features or use of this software
+      *    must display the following acknowledgement:
+      *    This product includes software developed by the University of
+      *    California, Berkeley and its contributors."
+
+Effective immediately, licensees and distributors are no longer required to
+include the acknowledgement within advertising materials.  Accordingly, the
+foregoing paragraph of those BSD Unix files containing it is hereby deleted
+in its entirety.
+
+William Hoskins
+Director, Office of Technology Licensing
+University of California, Berkeley

Makefile

@@ -2,7 +2,7 @@ include Makefile.config
 
 BIN=	kcat
 
-SRCS_y=	kcat.c format.c tools.c input.c
+SRCS_y=	kcat.c format.c tools.c input.c base64.c
 SRCS_$(ENABLE_JSON) += json.c
 SRCS_$(ENABLE_AVRO) += avro.c
 OBJS=	$(SRCS_y:.c=.o)

base64.c

@@ -0,0 +1,191 @@
+/*
+ * Base64 encoding/decoding (RFC 4648)
+ * Copyright (c) 2005-2019, Jouni Malinen <[email protected]>
+ * Copyright (c) 2019, Jeremy Lin <[email protected]>
+ *
+ * Based on:
+ * <https://github.com/freebsd/freebsd/blob/25d65ba7/contrib/wpa/src/utils/base64.c>
+ *
+ * - Removed the logic for adding line feeds every 72 chars, so this code is
+ *   no longer conformant to RFC 1341 (instead, RFC 4648).
+ *
+ * This software may be distributed under the terms of the BSD license.
+ * See LICENSE.base64 for more details.
+ */
+
+#include <stdlib.h>
+#include <string.h>
+
+#include "base64.h"
+
+static const unsigned char base64_table[65] =
+	"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+static const unsigned char base64_url_table[65] =
+	"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";
+
+size_t base64_encode_alloc_len(size_t len) {
+	return (len * 4 / 3 + 4) /* 3-byte blocks to 4-byte */
+                + 1; /* nul terminator */
+}
+
+static unsigned char* base64_gen_encode(const unsigned char *src, size_t len,
+					size_t *out_len,
+					const unsigned char *table,
+					int add_pad)
+{
+	unsigned char *out, *pos;
+	const unsigned char *end, *in;
+	const size_t olen = base64_encode_alloc_len(len);
+
+	if (olen < len)
+		return NULL; /* integer overflow */
+	out = malloc(olen);
+	if (out == NULL)
+		return NULL;
+
+	end = src + len;
+	in = src;
+	pos = out;
+	while (end - in >= 3) {
+		*pos++ = table[(in[0] >> 2) & 0x3f];
+		*pos++ = table[(((in[0] & 0x03) << 4) | (in[1] >> 4)) & 0x3f];
+		*pos++ = table[(((in[1] & 0x0f) << 2) | (in[2] >> 6)) & 0x3f];
+		*pos++ = table[in[2] & 0x3f];
+		in += 3;
+	}
+
+	if (end - in) {
+		*pos++ = table[(in[0] >> 2) & 0x3f];
+		if (end - in == 1) {
+			*pos++ = table[((in[0] & 0x03) << 4) & 0x3f];
+			if (add_pad)
+				*pos++ = '=';
+		} else {
+			*pos++ = table[(((in[0] & 0x03) << 4) |
+					(in[1] >> 4)) & 0x3f];
+			*pos++ = table[((in[1] & 0x0f) << 2) & 0x3f];
+		}
+		if (add_pad)
+			*pos++ = '=';
+	}
+
+	*pos = '\0';
+	if (out_len)
+		*out_len = pos - out;
+	return out;
+}
+
+static unsigned char* base64_gen_decode(const unsigned char *src, size_t len,
+					size_t *out_len,
+					const unsigned char *table)
+{
+	unsigned char dtable[256], *out, *pos, block[4], tmp;
+	size_t i, count, olen;
+	int pad = 0;
+	size_t extra_pad;
+
+	memset(dtable, 0x80, 256);
+	for (i = 0; i < sizeof(base64_table) - 1; i++)
+		dtable[table[i]] = (unsigned char) i;
+	dtable['='] = 0;
+
+	count = 0;
+	for (i = 0; i < len; i++) {
+		if (dtable[src[i]] != 0x80)
+			count++;
+	}
+
+	if (count == 0)
+		return NULL;
+	extra_pad = (4 - count % 4) % 4;
+
+	olen = (count + extra_pad) / 4 * 3;
+	pos = out = malloc(olen);
+	if (out == NULL)
+		return NULL;
+
+	count = 0;
+	for (i = 0; i < len + extra_pad; i++) {
+		unsigned char val;
+
+		if (i >= len)
+			val = '=';
+		else
+			val = src[i];
+		tmp = dtable[val];
+		if (tmp == 0x80)
+			continue;
+
+		if (val == '=')
+			pad++;
+		block[count] = tmp;
+		count++;
+		if (count == 4) {
+			*pos++ = (block[0] << 2) | (block[1] >> 4);
+			*pos++ = (block[1] << 4) | (block[2] >> 2);
+			*pos++ = (block[2] << 6) | block[3];
+			count = 0;
+			if (pad) {
+				if (pad == 1)
+					pos--;
+				else if (pad == 2)
+					pos -= 2;
+				else {
+					/* Invalid padding */
+					free(out);
+					return NULL;
+				}
+				break;
+			}
+		}
+	}
+
+	*out_len = pos - out;
+	return out;
+}
+
+/**
+ * base64_encode - Base64 encode
+ * @src: Data to be encoded
+ * @len: Length of the data to be encoded
+ * @out_len: Pointer to output length variable, or %NULL if not used
+ * Returns: Allocated buffer of out_len bytes of encoded data,
+ * or %NULL on failure
+ *
+ * Caller is responsible for freeing the returned buffer. Returned buffer is
+ * nul terminated to make it easier to use as a C string. The nul terminator is
+ * not included in out_len.
+ */
+char* base64_encode(const char *src, size_t len, size_t *out_len)
+{
+	return (char*) base64_gen_encode((const unsigned char*) src, len, out_len,
+                                         base64_table, 1);
+}
+
+char* base64_url_encode(const char *src, size_t len, size_t *out_len, int add_pad)
+{
+	return (char*) base64_gen_encode((const unsigned char*) src, len, out_len,
+                                         base64_url_table, add_pad);
+}
+
+/**
+ * base64_decode - Base64 decode
+ * @src: Data to be decoded
+ * @len: Length of the data to be decoded
+ * @out_len: Pointer to output length variable
+ * Returns: Allocated buffer of out_len bytes of decoded data,
+ * or %NULL on failure
+ *
+ * Caller is responsible for freeing the returned buffer.
+ */
+char* base64_decode(const char *src, size_t len, size_t *out_len)
+{
+	return (char*) base64_gen_decode((const unsigned char*) src, len, out_len,
+                                         base64_table);
+}
+
+char* base64_url_decode(const char *src, size_t len, size_t *out_len)
+{
+	return (char*) base64_gen_decode((const unsigned char*) src, len, out_len,
+                                         base64_url_table);
+}

base64.h

@@ -0,0 +1,27 @@
+/*
+ * Base64 encoding/decoding (RFC 4648)
+ * Copyright (c) 2005, Jouni Malinen <[email protected]>
+ * Copyright (c) 2019, Jeremy Lin <[email protected]>
+ *
+ * Based on:
+ * <https://github.com/freebsd/freebsd/blob/b53b2423/contrib/wpa/src/utils/base64.h>
+ *
+ * - Removed the logic for adding line feeds every 72 chars, so this code is
+ *   no longer conformant to RFC 1341 (instead, RFC 4648).
+ *
+ * This software may be distributed under the terms of the BSD license.
+ * See LICENSE.base64 for more details.
+ */
+
+#ifndef BASE64_H
+#define BASE64_H
+
+#include <stddef.h>
+
+size_t base64_encode_alloc_len(size_t len);
+char* base64_encode(const char *src, size_t len, size_t *out_len);
+char* base64_decode(const char *src, size_t len, size_t *out_len);
+char* base64_url_encode(const char *src, size_t len, size_t *out_len, int add_pad);
+char* base64_url_decode(const char *src, size_t len, size_t *out_len);
+
+#endif /* BASE64_H */