Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MEGA data processing errors #52

Open
philw6 opened this issue Jan 25, 2018 · 7 comments
Open

MEGA data processing errors #52

philw6 opened this issue Jan 25, 2018 · 7 comments

Comments

@philw6
Copy link

philw6 commented Jan 25, 2018 •
edited
Loading

There are a few errors that pop up when processing MEGA data, not sure if its only for this transducer set-up.

The first is that the horizontal distance of the output processed data is much less than actual. For instance the right bank is ~21m from the boat track in the attached file, where as PyHum is reading it as ~7m. bed_2picks0. This does not happen if I set the transducer to the incorrect "1199" (instead of "'mega'"), whereby the correct horizontal distance is returned.

Shadow removal does seem to remove anything beyond about 10m from the transducer where as the data was collected out to 30m. Contouring only returns a thin strip at the edge of the data area (see attached).
r00170class_contours0

The second issue is that I get a memory error in the gridding process where by it keeps trying larger and larger resolutions until infinity, regardless of having "99" or "1" in the res setting. I have to hit exit/enter in the CMD terminal to get it to stop. This also happens when I used the (incorrect) 1199 transducer type in the script.

I dont know if its relevant but I get the "memory mapping failed in sliding window - trying memory intensive version" in the initial read module. It seems to continue from this point OK though. Using a Windows 10 machine, i7-8550 with 16GB RAM.

Does anyone know the correct transducer array length for the MEGA units (XM 9 20 MSI T)? It physically measures about 16-18cm but I cant find details about the internal array length.
bed_pick0

Script below:

-- coding: utf-8 --

syntax for script without file path included: (pyhum) C:\Users\philw>python bhp_pyhum_pw1.py -i C:\Users\philw\BHPsonar\R00170.DAT -s C:\Users\philw\BHPsonar\R00170\

"""
Spyder Editor

This is a temporary script file.
"""
import sys, getopt

from Tkinter import Tk
from tkFileDialog import askopenfilename, askdirectory

import PyHum
import os

if name == 'main':

argv = sys.argv[1:]
humfile = ''; sonpath = ''

# parse inputs to variables
try:
   opts, args = getopt.getopt(argv,"hi:s:")
except getopt.GetoptError:
     print 'error'
     sys.exit(2)
for opt, arg in opts:
   if opt == '-h':
     print 'help'
     sys.exit()
   elif opt in ("-i"):
      humfile = arg
   elif opt in ("-s"):
      sonpath = arg

# prompt user to supply file if no input file given
if not humfile:
   print 'An input file is required!!!!!!'
   Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
   humfile = askopenfilename(filetypes=[("DAT files","*.DAT")]) 

# prompt user to supply directory if no input sonpath is given
if not sonpath:
   print 'A *.SON directory is required!!!!!!'
   Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
   sonpath = askdirectory() 

# print given arguments to screen and convert data type where necessary
if humfile:
   print 'Input file is %s' % (humfile)

if sonpath:
   print 'Son files are in %s' % (sonpath)
             
doplot = 1 #yes

# reading specific settings
cs2cs_args = "epsg:4326" #WGS84
bedpick = 1 # auto bed pick
c = 1450 # speed of sound fresh water
t = 0.108 # length of transducer for MEGA unit not confirmed
draft = 0.3 # draft in metres
flip_lr = 0 # flip port and starboard
model = 'mega' # humminbird model
calc_bearing = 0 #no
filt_bearing = 0 #no
chunk = 'd100' # distance, 100m
#chunk = 'p1000' # pings, 1000
#chunk = 'h10' # heading deviation, 10 deg
 
# correction specific settings
maxW = 1000 # rms output wattage for MEGA unit
dofilt = 0 # 1 = apply a phase preserving filter (WARNING!! takes a very long time for large scans)
correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections)
ph = 7.0 # acidity on the pH scale
temp = 26.0 # water temperature in degrees Celsius 20 at S4 dike and 27 at Mascarenhas in Oct 2017
salinity = 0.0 #salinity = 0.0


# for shadow removal
shadowmask = 0 #auto shadow removal
win =10

# for mapping
res = 99 # grid resolution in metres
# if res==99, the program will automatically calc res from the spatial res of the scans
mode = 1 # gridding mode (simple nearest neighbour)
#mode = 2 # gridding mode (inverse distance weighted nearest neighbour)
#mode = 3 # gridding mode (gaussian weighted nearest neighbour)
dowrite = 0 #disable writing of point cloud data to file
scalemax = 60 # max color scale value (60 is a good place to start)

## read data in SON files into PyHum memory mapped format (.dat)
PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, chunk) #cog

## correct scans and remove water column
PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity)

## remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline)
PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot)

## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
win = 10
numclasses = 4
PyHum.texture2(humfile, sonpath, win, doplot, numclasses)

## grid and map the scans
cs2cs_args = "epsg:4326"
res = 99
mode = 1
nn = 64
numstdevs = 4
use_uncorrected = 0
scalemax = 60
PyHum.map(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs, use_uncorrected, scalemax) #dowrite,
@dbuscombe-usgs
Copy link
Member

Interesting. Please provide this data set and I will investigate these issues. Thanks for your patience as these issues with the 'new' data formats get ironed out

@philw6
Copy link
Author

philw6 commented Jan 25, 2018

The .SON files are very large... can I send them to you via FTP link? To your email address perhaps?

@dbuscombe-usgs
Copy link
Member

sure thing. however you want to send is fine with me

@dbuscombe-usgs
Copy link
Member

I've fixed the issues associated with incorrect reported range, but I am also getting errors in the map module, because either the heading is incorrect or some other reason that is creating latitudes outside of the range -90 to +90. I haven't been able to figure it out. I've tried various epsg codes etc. Is there something unusual about your setup, such as an external non-Humminbird GPS?

@dbuscombe-usgs
Copy link
Member

Ok, I figured it out. For some reason, your data doesn't work with

epsg:4326

I managed to run the map module with

epsg:29101

using the latest version 1.4.6

@philw6
Copy link
Author

philw6 commented Feb 5, 2018 via email

@dbuscombe-usgs
Copy link
Member

Apologies for the delay in my response - the plotting routines in the texture module need an overhaul. I promise I will try to make it a priority soon. In the meantime, I suggest playing with the input parameter 'win', or just making your own plots with the exported data. Remember, PyHum is an API. You can incorporate it into your own scripts for making plots. I am planning a series of ipython notebooks that demonstrate some examples of this. Please bear with me - things progress slowly.

Also, you may use the e1e2 module in pyhum to compute E1 and E2 values. This is a little-used function within PyHum that I'd be happy to revise as directed

Always interested in collaboration - just let me know how I can help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dbuscombe-usgs @philw6