Notes
ArcGIS Online Clients
Post Date: Jan 3, 2023
Before moving on to another oppurtunity, I took time at my previous engagement to utilize ArcGIS Pro to access all GIS data that was present on a clients account and shared with me. I had set up seperate organizations in ArcGIS Online for eveyr client so that their data would be retian by them and there was no detangling of data should the client move on to another eninginer. I created shared groups for each client so that I could share data into and out of the group using my consultant account. This was before collaborations existed.
I thought it would be neat to see all of the data I had helped clients migrate to ArcGIS Online for the past several years. My computer ground to a halt with all the web requests and I didn't dare try to re-symbolize for clarity, so the raw dump of data is shown here. Cool to have interacted with such a variety of clients in teh region.
Final Flyby.aviPedestrian Bridge Model
Post Dec 10, 2015 2:07:46 PM
Sorting through some old files, I found this AutoCAD model that I created as an undergraduate student in 2009. I don't know if I've retained enough skill to recreate this. I wish I had retained some old Inventor models I created for fun even longer ago.
Tree Rootwad on Sketchfab
Post date: Nov 30, 2015 3:44:57 PM
This is a 3D model of the rootwad that was collected during my research at Pitt. The roots were cleaned then scanned with LIDAR. The resulting point cloud was cleaned and denoised in Meshlab and then the surface was created and cleaned up in Geomagic. The idea was to mount the roots to a hydraulic flume that was nearly complete in our lab and then compare physical measurements to a 3D model. Unfortunately, only some preliminary modeling was done and the flume was not completed in time.
Spreadsheet Contact Search
Post date: Sep 29, 2015 2:36:54 PM
I keep my contacts on Google where it is easy to search and find a phone number that I need. I missed this ease of use for finding contacts at work. I built a quick DIY solution that several people could use to keep track of shared contacts and their info, without paying for CRM software. This isn't unique, I borrowed from advice on this site [1] and modified to make it looks nice and fit my needs. It's not fancy but I find it useful to search for contacts by company for county they are responsible for, etc..
You can use the same setup in Google Sheets and it would be easy to setup a Google Form to feed contact info into the table. The one downside is Sheets doesn't have support for tables like Excel does, so you might have to write some script to be sure all the hidden concatenation fields are copied down to new rows. At that point it might be easier to set up an Awesome Table.
To get started, set up a tableview in a tab in excel. Add any fields that you'd like. Column A is hidden and uses the RANK function to rank the score given by using MATCH and INDEX in column Q. I like that I have lots of flexibility in what fields I want.
You can see the ranks in column Q, rows that don't contain the search string are not ranked, so the search isn't smart like a Google search. I combine the fields that I want to search in column R as shown in the formula bar. I then compile several columns that format the info into multiple lines to display nicely.
Performing a search returns ranked items (using vlookup and numbers in column A that are colored white to match the background). I also added a modifier to search by in case I am getting results that I don't intend. i could just search for previous projects, for example.
Working with SSURGO Soil Data
Post date: Sep 22, 2015 3:26:32 PM
Most soil data can be viewed and downloaded from the Web Soil Survey here:
http://websoilsurvey.sc.egov.usda.gov/App/HomePage.htm
However, the website needs some improvement and I can't always find what I am looking for. Furthermore the way that the soil data is stored is complicated. I am no expert on databases, so I'm sure there is some efficiency reason for this, but basically various group of properties are stored across several tables that have complex relations to each other. Previously, I had downloaded soil data for a few counties that I worked in and then joined the appropriate tables in ArcGIS to be able to display what I wanted. This was cumbersome.
My solution was to download all soil data for the state and surrounding areas. You can do this through the NRCS website if you choose to download by drawing a region and simply draw a huge rectangle across the state. The site will then let you download the data. Unfortunately this data is stored at the county level. So you end up with dozens of folders for the state, each with the shapefiles and data tables for that county. Ideally, I'd like a single folder. But after some frustrating attmepts to combine everything, I gave up and used the Soil Data Viewer available here:
http://www.nrcs.usda.gov/wps/portal/nrcs/detailfull/soils/home/?cid=nrcs142p2_053620
To use this tool, add the area shapefile for the county of interest to ArcGIS, then you have to ensure that the database is properly linked. You can do this by opening the database in Access and pointing it to the tabular folder. Then you can select the database in the Soil Data Viewer and spit out any number of properties as shapefiles.
I compiled all hydrologic soil groups for Pennsylvania and saved that as a layer file and a tile package. The layer is what I use programmatically to pull hydrologic soil class from and the tile package is for viewing and creating maps since it's easy to work with but contains no actual data.
Lastly, since all soil shapefiles are in separate folders, I wrote a small recursive python script to add all shapefiles based on some criteria, like all of Ohio. (below)
Eventually I want to figure out a more elegant solution. I know one is possible based on the EPA Stormwater Calculator. It let's you grab soil hydrologic data directly from the web. I would love to see how this works. Someday..
import os
import sys
rootDir = 'C:\\Users\\user\\Documents\\ArcGIS Custom\\GIS Data\\NRCS Soils'
mxd = arcpy.mapping.MapDocument("CURRENT")
df = arcpy.mapping.ListDataFrames(mxd,"*")[0]
print(rootDir)
for dirName, subdirList, fileList in os.walk(rootDir):
for fname in fileList:
if 'soilmu_a_' in fname and '.shp' in fname and 'oh' in fname:
print('\tFound %s' % fname)
newlayer = arcpy.mapping.Layer(dirName + '\\' + fname)
arcpy.mapping.AddLayer(df, newlayer,"BOTTOM")
GDAL to Extract Custom tiffs from GeoPDFs
Post date: May 29, 2015 3:50:39 PM
A co-worker was using geoPDFs from the USGS and needed to extract tiffs that were of a reasonable size. He mentioned that GDAL was a package that seemed to be used to do this commonly. I hadn't heard of it but this was clearly a cool tool to learn more about. It turns out that GDAL is built into QGIS which is great, except I wasn't able to figure out how to configure some of the options to my liking. I ended up installing GDAL separately from QGIS and writing a small script that I can edit and run to execute a command line. Here's how I did it:
This document is a great summary of how GDAL commands can produce compressed and customized tiffs from geoPDFS.
This was a helpful page to get GDAL installed properly. I had some issues with paths and dlls, I believe this was from QGIS being installed or perhaps because I have python 2 and 3 installed? I'm not sure. I ended up uninstalling ArcGIS and QGIS, then installing GDAL and adding the paths, then reinstalling everything.
Here's where you download packaged GDAL and lastly here is reference for some of the command options.
I wrote a small file (attached) that isn't user friendly but I'd rather make edits in notepad then type things out in command line (until Windows 10 releases and we get copy/paste in cmd)
GDAL is super useful because you can extract only the layers you want and produce a very reasonably sized file using compression in a reasonably short time. Testing different options, I found it best to remove the alpha channel then compress to jpeg with at least 75 quality (default) and a dpi of 200-300. But that all depends on what you're using it for of course.
An original tiff unmodified was 927MB, removing alpha reduced to 695MB, exporting at 300 dpi and highest compression quality was down to 38MB and reducing the quality to 75 resulted in just 10MB.
