Skip to content

Commit

Permalink
ZAPD update: libpng, zroom improvements and others (zeldaret#811)
Browse files Browse the repository at this point in the history
* git subrepo pull --force tools/ZAPD

subrepo:
  subdir:   "tools/ZAPD"
  merged:   "769f5702a"
upstream:
  origin:   "https://github.com/zeldaret/ZAPD.git"
  branch:   "master"
  commit:   "769f5702a"
git-subrepo:
  version:  "0.4.3"
  origin:   "???"
  commit:   "???"

* Add `libpng` to readme

* Remove `-ifp` since it doesn't exists anymore in ZAPD

* Remove extra print I added

* Add UNK_09 macro and other minor fixes

* Simplify PNG rules

* simplify gitignore

* Update README.md

Co-authored-by: Roman971 <[email protected]>

* Update dockerfile

* basic instructions for cygwin and mac

* git subrepo pull --force tools/ZAPD

subrepo:
  subdir:   "tools/ZAPD"
  merged:   "86160be69"
upstream:
  origin:   "https://github.com/zeldaret/ZAPD.git"
  branch:   "master"
  commit:   "86160be69"
git-subrepo:
  version:  "0.4.3"
  origin:   "???"
  commit:   "???"

* Change nanoseconds to seconds in extract_assets.py

Co-authored-by: Roman971 <[email protected]>
  • Loading branch information
2 people authored and louist103 committed Aug 2, 2021
1 parent e72e24e commit 7ecc221
Show file tree
Hide file tree
Showing 142 changed files with 5,828 additions and 14,641 deletions.
11 changes: 3 additions & 8 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,10 @@ RUN apt-get update && \
unzip \
clang-tidy \
clang-format \
clang-format-11 \
nano \
vbindiff

RUN wget \
https://github.com/n64decomp/qemu-irix/releases/download/v2.11-deb/qemu-irix-2.11.0-2169-g32ab296eef_amd64.deb \
-O qemu.deb && \
echo 8170f37cf03a08cc2d7c1c58f10d650ea0d158f711f6916da9364f6d8c85f741 qemu.deb | sha256sum --check && \
dpkg -i qemu.deb && \
rm qemu.deb
vbindiff \
libpng-dev

RUN python3 -m pip install --user colorama ansiwrap attrs watchdog python-Levenshtein
RUN python3 -m pip install --upgrade attrs pycparser
Expand Down
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,12 +55,13 @@ For macOS, use homebrew to install the following dependencies:
* make
* python3
* md5sha1sum
* libpng

You can install them with the following commands:

```bash
brew update
brew install coreutils make python3 md5sha1sum
brew install coreutils make python3 md5sha1sum libpng
```

You'll also need to [build and install mips-linux-binutils](docs/BUILDING_BINUTILS_MACOS.md).
Expand All @@ -80,12 +81,13 @@ The build process has the following package requirements:
* build-essential
* binutils-mips-linux-gnu
* python3
* libpng-dev

Under Debian / Ubuntu (which we recommend using), you can install them with the following commands:

```bash
sudo apt-get update
sudo apt-get install git build-essential binutils-mips-linux-gnu python3
sudo apt-get install git build-essential binutils-mips-linux-gnu python3 libpng-dev
```

#### 2. Clone the repository
Expand Down Expand Up @@ -142,7 +144,6 @@ This means that the built ROM isn't the same as the base one, so something went

Both of these have the disadvantage that the ordering of the terminal output is scrambled, so for debugging it is best to stick to one thread (i.e. not pass `-j` or `-jN`).


### Cygwin

If you want to use Cygwin, you will need to:
Expand All @@ -156,6 +157,7 @@ Once mips-linux-binutils is installed you will need to install the following pac
* libiconv
* dos2unix
* python3
* libpng-devel

Then you can continue from step [step 2](#2-clone-the-repository) of the Linux instructions.

Expand Down
2 changes: 1 addition & 1 deletion assets/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@
*.h
*.cfg
*.vtx.inc
*.dlist.inc
*.dlist.inc
2 changes: 1 addition & 1 deletion assets/xml/scenes/test_levels/syotes.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
<Scene Name="syotes_scene" Offset="0x0"/>
</File>
<File Name="syotes_room_0" Segment="3">
<Room Name="syotes_room_0" Offset="0x0"/>
<Room Name="syotes_room_0" HackMode="syotes_room" Offset="0x0"/>
</File>
</Root>
21 changes: 4 additions & 17 deletions extract_assets.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,12 @@ def SignalHandler(sig, frame):
mainAbort.set()
# Don't exit immediately to update the extracted assets file.

def Extract(xmlPath, outputPath, outputSourcePath):
ExtractFile(xmlPath, outputPath, outputSourcePath, 1, 0)

def ExtractScene(xmlPath, outputPath, outputSourcePath):
ExtractFile(xmlPath, outputPath, outputSourcePath, 1, 1)

def ExtractFile(xmlPath, outputPath, outputSourcePath, genSrcFile, incFilePrefix):
def ExtractFile(xmlPath, outputPath, outputSourcePath):
if globalAbort.is_set():
# Don't extract if another file wasn't extracted properly.
return

execStr = "tools/ZAPD/ZAPD.out e -eh -i %s -b baserom/ -o %s -osf %s -gsf %i -ifp %i -rconf tools/ZAPDConfigs/MqDbg/Config.xml" % (xmlPath, outputPath, outputSourcePath, genSrcFile, incFilePrefix)
execStr = "tools/ZAPD/ZAPD.out e -eh -i %s -b baserom/ -o %s -osf %s -gsf 1 -rconf tools/ZAPDConfigs/MqDbg/Config.xml" % (xmlPath, outputPath, outputSourcePath)
if globalUnaccounted:
execStr += " -wu"

Expand All @@ -45,10 +39,6 @@ def ExtractFunc(fullPath):
outPath = os.path.join("assets", *pathList[2:], objectName)
outSourcePath = outPath

isScene = fullPath.startswith("assets/xml/scenes/")
if isScene:
objectName += "_scene"

if not globalForce:
if fullPath in globalExtractedAssetsTracker:
timestamp = globalExtractedAssetsTracker[fullPath]["timestamp"]
Expand All @@ -59,10 +49,7 @@ def ExtractFunc(fullPath):

currentTimeStamp = int(time.time())

if isScene:
ExtractScene(fullPath, outPath, outSourcePath)
else:
Extract(fullPath, outPath, outSourcePath)
ExtractFile(fullPath, outPath, outSourcePath)

if not globalAbort.is_set():
# Only update timestamp on succesful extractions
Expand Down Expand Up @@ -96,7 +83,7 @@ def main():

extractedAssetsTracker = manager.dict()
if os.path.exists(EXTRACTED_ASSETS_NAMEFILE):
with open(EXTRACTED_ASSETS_NAMEFILE) as f:
with open(EXTRACTED_ASSETS_NAMEFILE, encoding='utf-8') as f:
extractedAssetsTracker.update(json.load(f, object_hook=manager.dict))

asset_path = args.single
Expand Down
4 changes: 2 additions & 2 deletions tools/ZAPD/.gitrepo
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
[subrepo]
remote = https://github.com/zeldaret/ZAPD.git
branch = master
commit = b9120803e6094a54192ed67d9585ab046101c816
parent = 81b43ba32e564a00c976dec7d520e6bcb0b122f8
commit = 86160be6968c9813fcf359b88a0ac188d87c2772
parent = 07c93fd180beb092cb0f50faae04c57c0eff458a
method = merge
cmdver = 0.4.3
38 changes: 28 additions & 10 deletions tools/ZAPD/Makefile
Original file line number Diff line number Diff line change
@@ -1,27 +1,40 @@
OPTIMIZATION_ON ?= 1
ASAN ?= 0
DEPRECATION_OFF ?= 0
CFLAGS ?=

CC := g++
INC := -I ZAPD -I lib/assimp/include -I lib/elfio -I lib/json/include -I lib/stb -I lib/tinygltf -I lib/libgfxd -I lib/tinyxml2
CFLAGS += -g3 -ggdb -fpic -std=c++17 -rdynamic -Wall -fno-omit-frame-pointer

CFLAGS := -g3 -fpic -Wl,-export-dynamic -std=c++17 -rdynamic -Wall
ifeq ($(OPTIMIZATION_ON),1)
CFLAGS += -O2
else
ifeq ($(OPTIMIZATION_ON),0)
CFLAGS += -O0
else
CFLAGS += -O2
endif
ifneq ($(ASAN),0)
CFLAGS += -fsanitize=address
endif
ifneq ($(DEPRECATION_OFF),0)
CFLAGS += -DDEPRECATION_OFF
endif
# CFLAGS += -DTEXTURE_DEBUG

LDFLAGS := -ldl
LDFLAGS := -ldl -lpng
UNAME := $(shell uname)

FS_INC =
FS_INC ?=
ifneq ($(UNAME), Darwin)
FS_INC += -lstdc++fs
CFLAGS += -Wl,-export-dynamic
endif

SRC_DIRS := ZAPD ZAPD/ZRoom ZAPD/ZRoom/Commands ZAPD/Overlays ZAPD/HighLevel ZAPD/OpenFBX
SRC_DIRS := ZAPD ZAPD/ZRoom ZAPD/ZRoom/Commands ZAPD/Overlays ZAPD/HighLevel

ZAPD_CPP_FILES := $(foreach dir,$(SRC_DIRS),$(wildcard $(dir)/*.cpp))
ZAPD_H_FILES := $(foreach dir,$(SRC_DIRS),$(wildcard $(dir)/*.h))

CPP_FILES := $(foreach dir,$(SRC_DIRS),$(wildcard $(dir)/*.cpp))
CPP_FILES += lib/tinyxml2/tinyxml2.cpp
CPP_FILES += $(ZAPD_CPP_FILES) lib/tinyxml2/tinyxml2.cpp
O_FILES := $(CPP_FILES:.cpp=.o)

all: ZAPD.out copycheck
Expand All @@ -38,14 +51,19 @@ clean:

rebuild: clean all

format:
clang-format-11 -i $(ZAPD_CPP_FILES) $(ZAPD_H_FILES)

.PHONY: all genbuildinfo copycheck clean rebuild format

%.o: %.cpp
$(CC) $(CFLAGS) $(INC) -c $< -o $@ $(LDFLAGS)

ZAPD/Main.o: genbuildinfo ZAPD/Main.cpp
$(CC) $(CFLAGS) $(INC) -c ZAPD/Main.cpp -o $@ $(LDFLAGS)

lib/libgfxd/libgfxd.a:
$(MAKE) -C lib/libgfxd -j
$(MAKE) -C lib/libgfxd

ZAPD.out: $(O_FILES) lib/libgfxd/libgfxd.a
$(CC) $(CFLAGS) $(INC) $(O_FILES) lib/libgfxd/libgfxd.a -o $@ $(FS_INC) $(LDFLAGS)
103 changes: 103 additions & 0 deletions tools/ZAPD/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# ZAPD: Zelda Asset Processor for Decomp

## Compiling

### Dependencies

ZAPD needs a compiler with C++17 support.

ZAPD has the following library dependencies:

- `libpng`

In a Debian/Ubuntu based environment, those could be installed with the following command:

```bash
sudo apt install libpng-dev
```

### Building

#### Linux / *nix

ZAPD uses the clasic `Makefile` approach. To build just run `make` (or even better `make -j` for faster compilations).

You can configure a bit your ZAPD build with the following options:

- `OPTIMIZATION_ON`: If set to `0`, then optimizations will be disabled (compile with `-O0`). Any other value compiles with `-O2`. Defaults to `1`.
- `ASAN`: If it is set to a non-zero then ZAPD will be compiled with Address Sanitizer enabled (`-fsanitize=address`). Defaults to `0`.
- `DEPRECATION_OFF`: If it is set to a non-zero then deprecation warnings will be disabled. Defaults to `0`.

As an example, if you want to build ZAPD with optimizations disabled and use the address sanitizer, you could use the following command:

```bash
make -j OPTIMIZATION_ON=0 ASAN=1
```

#### Windows

This repository contains `vcxproj` files for compiling under Visual Studio environments. See `ZAPD/ZAPD.vcxproj`.

## Invoking ZAPD

ZAPD needs a _File parsing mode_ to be passed as first parameter. The options are:

- `e`: "Extraction" mode.
- In this mode, ZAPD expects a XML file as input, a folder as ouput and a path to the baserom files.
- ZAPD will read the XML and use it as a guide to extract the contents of the specified asset file from the baserom folder.
- For more info of the format of those XMLs, see the [ZAPD extraction XML reference](docs/zapd_extraction_xml_reference.md).
- `bsf`: "Build source file" mode.
- This is an experimental mode.
- It was going to be used to let you have XMLs that aren't just for extraction. Might get used, might not. Still need to experiment on that.
- `btex`: "Build texture" mode.
- In this mode, ZAPD expects a PNG file as input, a filename as ouput and a texture type parameter (`-tt`).
- ZAPD will try to convert the given PNG into the contents of a `uint64_t` C array.
- `bren`: "Build (render) background" mode.
- In this mode, ZAPD expects a JPG file as input and a filename as ouput.
- ZAPD will try to convert the given JPG into the contents of a `uint64_t` C array.
- `blb`: "Build blob" mode.
- In this mode, ZAPD expects a BIN file as input and a filename as ouput.
- ZAPD will try to convert the given BIN into the contents of a `uint8_t` C array.
- `bovl`: "Build overlay" mode.
- In this mode, ZAPD expects an overlay C file as input, a filename as ouput and an overlay configuration path (`-cfg`).
- ZAPD will generate a reloc `.s` file.

ZAPD also accepts the following list of extra parameters:

- `-i PATH` / `--inputpath PATH`: Set input path.
- `-o PATH` / `--outputpath PATH`: Set output path.
- `-b PATH` / `--baserompath`: Set baserom path.
- Can be used only in `e` or `bsf` modes.
- `-osf PATH`: Set source output path. This is the path where the `.c` and `.h` files will be extracted to. If omitted, it will use the value passed to `--outputpath` parameter.
- `-gsf MODE`: Generate source file during extraction. If `MODE` is `1`, C source files will be generated.
- Can be used only in `e` mode.
- `-crc` / `--output-crc`: Outputs a CRC file for each extracted texture.
- Can be used only in `e` or `bsf` modes.
- `-ulzdl MODE`: Use "Legacy ZDisplayList" instead of `libgfxd`. Set `MODE` to `1` to enable it.
- Can be used only in `e` or `bsf` modes.
- `-profile MODE`: Enable profiling. Set `MODE` to `1` to enable it.
- `-uer MODE`: Split resources into their individual components (enabled by default). Set `MODE` to non-`1` to disable it.
- `-tt TYPE`: Set texture type.
- Can be used only in mode `btex`.
- Valid values:
- `rgba32`
- `rgb5a1`
- `i4`
- `i8`
- `ia4`
- `ia8`
- `ia16`
- `ci4`
- `ci8`
- `-cfg PATH`: Set cfg path (for overlays).
- Can be used only in `bovl` mode.
- `-rconf PATH` Read Config File.
- `-eh`: Enable error handler.
- Only available in non-Windows environments.
- `-v MODE`: Enable verbosity. Currently there are 3 possible values:
- `0`: Default. Completely silent (except for warnings and errors).
- `1`: Information.
- `2` (and higher): Debug.
- `-wu` / `--warn-unaccounted`: Enable warnings for each unaccounted block of data found.
- Can be used only in `e` or `bsf` modes.
- `-tm MODE`: Test Mode (enables certain experimental features). To enable it, set `MODE` to `1`.
79 changes: 79 additions & 0 deletions tools/ZAPD/ZAPD/Declaration.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
#include "Declaration.h"

Declaration::Declaration(DeclarationAlignment nAlignment, DeclarationPadding nPadding, size_t nSize,
std::string nText)
{
alignment = nAlignment;
padding = nPadding;
size = nSize;
text = nText;
}

Declaration::Declaration(DeclarationAlignment nAlignment, size_t nSize, std::string nVarType,
std::string nVarName, bool nIsArray, std::string nText)
: Declaration(nAlignment, DeclarationPadding::None, nSize, nText)
{
varType = nVarType;
varName = nVarName;
isArray = nIsArray;
}

Declaration::Declaration(DeclarationAlignment nAlignment, DeclarationPadding nPadding, size_t nSize,
std::string nVarType, std::string nVarName, bool nIsArray,
std::string nText)
: Declaration(nAlignment, nPadding, nSize, nText)
{
varType = nVarType;
varName = nVarName;
isArray = nIsArray;
}

Declaration::Declaration(DeclarationAlignment nAlignment, size_t nSize, std::string nVarType,
std::string nVarName, bool nIsArray, size_t nArrayItemCnt,
std::string nText)
: Declaration(nAlignment, DeclarationPadding::None, nSize, nText)
{
varType = nVarType;
varName = nVarName;
isArray = nIsArray;
arrayItemCnt = nArrayItemCnt;
}

Declaration::Declaration(DeclarationAlignment nAlignment, size_t nSize, std::string nVarType,
std::string nVarName, bool nIsArray, std::string nArrayItemCntStr,
std::string nText)
: Declaration(nAlignment, DeclarationPadding::None, nSize, nText)
{
varType = nVarType;
varName = nVarName;
isArray = nIsArray;
arrayItemCntStr = nArrayItemCntStr;
}

Declaration::Declaration(DeclarationAlignment nAlignment, size_t nSize, std::string nVarType,
std::string nVarName, bool nIsArray, size_t nArrayItemCnt,
std::string nText, bool nIsExternal)
: Declaration(nAlignment, nSize, nVarType, nVarName, nIsArray, nArrayItemCnt, nText)
{
isExternal = nIsExternal;
}

Declaration::Declaration(DeclarationAlignment nAlignment, DeclarationPadding nPadding, size_t nSize,
std::string nVarType, std::string nVarName, bool nIsArray,
size_t nArrayItemCnt, std::string nText)
: Declaration(nAlignment, nPadding, nSize, nText)
{
varType = nVarType;
varName = nVarName;
isArray = nIsArray;
arrayItemCnt = nArrayItemCnt;
}

Declaration::Declaration(std::string nIncludePath, size_t nSize, std::string nVarType,
std::string nVarName)
: Declaration(DeclarationAlignment::None, DeclarationPadding::None, nSize, "")
{
includePath = nIncludePath;
varType = nVarType;
varName = nVarName;
}
Loading

0 comments on commit 7ecc221

Please sign in to comment.