🌱 Google Earth Engine 101

mapshaper

Ok. 

I wanna know why have I never heard of this online tool before. Like, what the hell is wrong with the social media? Is something wrong with Twitter or Instagram or something that they never caught on mapshaper? Or was it just me and my hazardous ignorance, yet again?

Have you tried this free nifty online tool that literally simplify crazy complicated shapefile polygons like it’s no one’s business?!

It started with some last minute inspiration on how to collate data from 3 different regions; developed from remote sensing techniques which vary from one another. The common output here is to turn all of them into a vector file; namely shapefile, and start working on the attribute to ease merging of the different shapefile layers.

Once merged, this shapefile is to be published as a hosted feature layer into the ArcGIS Online platform and incorporated into a webmap that serves as a reference data to configure/design a dashboard. What is a dashboard? It's basically an app template in ArcGIS Online that summarizes all the important information in your spatial data. It's a fun app to create, no coding skills required. Check out the gallery here for reference:

Operations Dashboard for ArcGIS Gallery

There are two common ways to publish hosted feature layer into ArcGIS Online platform.

Method 1: Zip up the shapefile and upload it as your content. This will trigger the command inquiring if you would like to publish it as a hosted feature layer. You click 'Yes' and give it a name and et voila! You have successfully publish a hosted feature layer.

Method 2: From an ArcGIS Desktop or ArcGIS Pro, you publish them as feature service (as ArcMap calls them) or web layer (as the its sister ArcGIS Pro calls them). Fill up the details and enabling the function then hit 'Publish' and it will be in the platform should there be no error or conflicting issues.

So, what was the deal with me and mapshaper? 

🛑 A fair warning here and please read these bullet points very carefully:

I need you to remember...I absolve any responsibility of what happens to your data should you misinterpreted the steps I shared. 

Please always  👏🏻  BACK 👏🏻 UP  👏🏻  YOUR 👏🏻 DATA. Don’t even try attempting any tools or procedure that I am sharing without doing so. Please. Cause I am an analyst too and hearing someone else forget to save their data or create a backup is enough to make me die a little inside. 

For this tool, please export out the attribute table of your shapefile because this tool will CHANGE YOUR SHAPEFILE ATTRIBUTES. 

When I was publishing the vector I have cleaned and feature-engineered via ArcGIS Pro...it took so long that I was literally dying inside. I'm not talking about 20 minutes or an hour. It took more than 12 hours and it did not conjure the 'Successfully published' notification as I would've expected from it.

So at around 5.30 am, I randomly type 'simplify shapefily online free'. Lo and behold, there was mapshaper.

All I did was, zip up my polygon, drag it to the homepage and it will bring you to the option of choosing the actions that will be executed while the data is being imported into mapshaper:

detect line intersections

snap vertices

This option will help you to detect the intersections of lines within your vector/shapefile. This can help identify topological error.

The option to snap vertices will snap together points of similar or almost identical coordinate system. But it does not work with TopoJSON formats.

There is something interesting about this options too; you can enter other types of customized options provided by the tool from its command line interface! But hold your horses peeps. I did not explore that because here, we want to fix an issue and we'll focus on that first. I checked both options and import them in.

This will bring the to a page where there you can start configuring options and method to simplify your vector.

To simplify your shapefile, you can have both options to prevent the shape of the polygon being compromised; prevent shape removal, and to utilize the planar Cartesian geometry instead of the usual geoid longitude and latitude; use planar geometry. The implication of the second option is not obvious to me yet since all I wanted was to get the data simplified for easy upload and clean topology, thus, I chose both options to maintain the shape and visibility of all my features despite the highest degree of simplification.

Alike to the options of methodology for simplication in the mainstream software, I can see familiar names:

Douglas-Peuker

Visvalingam / effective area

Visvalingam / weighted area

First and foremost, I had no slightest idea of what these were. Like for real. I used to just go first for the default to understand what sort of output it will bring me. But here, the default; Visvalingam / weighted area, seemed like the best option. What are these methodologies of simplification? There are just algorithms used to help simplify your vectors:

🎯 Douglas-Peucker algorithm decimates a curve composed of line segments to a similar curve with fewer points (Ramer-Douglas-Peucker algorithm, Wikipedia; 2021).

🎯 Visvalingam algorithm is a line simplication operator that works eliminating any less significant points of the line based on effective area concept. That basically means that the triangle formed by each of the line points with two of its immediate neighboring points (Visvalingam Algorithm | aplitop).

🎯 Visvalingam algorithm with weight area is another version of Visvalingam algorithm of subsequent development where an alternative metrics is used and weighted to take into account the shape (Visvalingam & Whelan, 2016).

For reasons I can't even explain, I configured my methodology to utilize the third option and now that I have the time to google it, Thank God I did.

Then, see and play with the magic at the 'Settings' slider where you can adjust and view the simplification made onto the vector! I adjusted it to 5%. The shape retained beautifully. And please bear in mind, this vector was converted from a raster. So, what I really wanted is the simplified version of the cleaned data and to have them uploaded.

Now that you've simplified it, export it into a zipped folder of shapefile and you can use it like any other shapefile after you extracted it.

Remember when I say you have got to export your table of attributes out before you use this tool? Yea...that's the thing. The attribute table will shock you cause it'll be empty. Literally. With only the OBJECTID left. Now, with that attribute table you've backed up, use the 'Join Table' tool in ArcGIS Pro or ArcMap and join the attribute table in without any issues.

Phewh!!

I know that it has alot more functions than this but hey, I'm just getting started. Have you ever done anything more rocket science than I did like 2 days ago, please share it with the rest of us. Cause I gotta say, this thing is cray!! Love it so much.

mapshaper developer, if you're seeing this, I 🤟🏻 you!

UPDATE

I have been asked about the confidentiality of the data. I think this is where you understand the reason behind the fact that they will work even with using just the ‘.shp’ file of the shapefile since _that_ is the vector portion of the shapefile. 

Shapefile is a spatial data format that is actually made up of 4 files; minimum. Each of these files share the same name with different extensions; .prj, .shx, .shp and .dbf. Although I am not familiar with what .shx actually accounts for, the rest of them are pretty straightforward:

.prj: stores the projection information

.dbf: stores the tabulated attributes of each features in the vector file

.shp: stores the shape/vector information of the shapefile. 

So, as the tool indicate, it actually helps with the vector aspect of your data which is crucial in cartography. 

diy geo app

Esri has been releasing more and more MOOC over the span of 2 years to accommodate its increasingly large expanse of products within the ArcGIS ecosystem. 

But of all the MOOCs that I've participated in, 'Do-It-Yourself Geo App MOOC' must be the most underrated ones produced by Esri Training. The functionalities highlighted within the MOOC took the anthem right off their recent Esri UC 2020 that went virtual. The curriculum includes:

The creation of hosted feature layer (without utilizing any GIS software medium like ArcMap or ArcGIS Pro).

The basics of the ArcGIS Online platform ecosystem:

hosted feature layer >  web map > web app 

Basically, to view a hosted feature layer, you will need to drag it onto a 'Map' and save it as a web map.

Conventionally, web map suffices for the visualization and analytical work for the likes of any geospatialist who are familiar with Web GIS. 

But this time, Esri is highlighting a brand new web map product called 'Map Viewer Beta'. Why beta? Cause it is still in beta version but so sleeky cool that they just had to let every have a shot at using it. Truth be told, Map Viewer Beta did not disappoint.

Even so, Map Viewer Beta still has some functionalities that have yet to be implemented. 

Using web map to visualize data, configure pop-up, execute simple analysis and extending it to Map Viewer Beta interface 

Utilizing Survey123 for crowdsourcing data; the first level of citizen science and creating a webmap out of it.

Creating native apps using AppStudio for ArcGIS; no coding required. 

Some tidbits on accessing the ArcGIS API for JavaScript

I love how cool it is that this MOOC actually shows you step-by-step on how to use the new Map Viewer Beta and explain the hierarchy of formats for the published content in the ArcGIS Online platform

I have established my understanding of ArcGIS Online ecosystem 3 years back but I do find it awkward that such powerful information is not actually summarized in a way that is comprehensible for users that have every intention of delving into Web GIS. And Web GIS is the future with all the parallel servers that could handle the processing/analysis of large amount of data. ArcGIS Online is a simplified platform that provides interfaces for the fresh-eyed new geospatial professionals. 

It is quite well-know for the fact that there has been some criticism as to the domination of Esri within the GIS tools/resources within the geospatial science industry, but I believe it is something we could take as a pinch of salt. Not everything in Esri's massive line of commercial products are superior to other platforms but it is a starting point for any new geospatialists who wants to explore technologies there are not familiar with. 

All in all, this MOOC is heaven-sent. For me, I have been playing with the web apps and web maps for close to 4 years and I can attest to the fact that it covers all the basics. For the developer's bit, maybe not so much as going through it in a distinct step-by-step but it does stoke the curiosity as to how it works. The question is, how do we make it work. Now that's a mystery I am eager to solve. 

I'm going to put this on my ever-expanding to-do list and think JavaScript for another few more months of testing out this ArcGIS API for JavaScript implementation. Tell me if you wanna know how this actually works and I'll share what I find out when I do.

For those who had missed out on this cohort, fear not. This MOOC runs twice a year and the next cohort is going to be from Feb 17 to March 17 2021. The registration is already open, so don’t hold back and click the link below:

Do-It-Yourself Geo Apps

Do register for a public account before signing up or just click 'Register' at the MOOC's page and it's open the open to either sign in or 'Create a public account'. It was a blast and I'm sure, if you've never used any of the feature I've mentioned above, you'll be as wide-eyed as I was 3 years ago. :D

Till then, stay spatially mappy comrades!

P/S: If you complete all the assignments and quizzes, you'll get a certificate of completion from Esri. Which is pretty rad!

Don't break the chain peeps! Reblog cause I'm looking for inspiration for my next masterpiece! 🙇🙇🙇

reblog/like if you’re an active studyblr/langblr

I’ve just unfollowed a bunch of inactive blogs, now that I follow ONLY 54 blogs??? pls reblog/like so I can have an active dashboard and new friends hehehe

🗻 Diorama Drama

So, John Nelson made this absolutely simple tutorial on how to create a diorama interesting features on Earth's surfaces. I have been eyeing alot of people making them and resisting the urge to make one. I mean, I have the habit of falling into a rabbit hole with making maps that I'll definitely be having too much fun to stop. That could mean hours and hours scrutinizing colors, perspective or setups and even legend arrangements. But this...I decided not to overdo it and just make one.

I can't believe how easy it is to make one! Here's a piece of many to come; the elevation diorama of area within the Batang Ai Dam and Batang Ai National Park in Sarawak, East Malaysia.

What you'll learn... 1️⃣ How to extract raster from existing image layer in Living Atlas; that's how we extracted the DEM layer for this elevation data.

2️⃣ How to create a 3D scene in ArcGIS Pro and navigate through them.

3️⃣ How to create a layout layer and export them as a static image format i.e jpeg, png, tif etc...

Check out the video tutorial here 👇🏻

I was thinking that this tutorial is making me feel like there is a possibility of creating some floating island or pixel-like models of isolated pieces of land -- my head is imagining some kind of dragons flying around in some nonexistent Viking fantasy map. But hey...if you're not inspired to create crazier maps from John's tutorial, then are you even a cartographer at all?

He's on Twitter, LinkedIn, YouTube and he even has this supercool blog of his; Adventures in Mapping, where he shares all of his tutorials in full documentation and gorgeous portfolio that could only consist of magic! One such as I can only aspire.

So, that was what our Sunday looked like and I think I'll be playing around with dioramas for a few weeks. Let's see just how many of these I can make 'til the next Sunday.

Malaysia Bid Round 2023 (MBR 2023)

Tool: ArcGIS Pro 2.6.3 Technique: Symbolization, labeling and SQL expression

MBR 2023 is a peak event that culminates all the effort of data collection and stock take of hydrocarbon resource in the Malaysia. It is an annual event that put together all the exploration blocks, discoverable hydrocarbon fields and late life assets for upstream sectors to evaluate and invest in.

Leading up to the event, the Malaysia Petroleum Management (MPM) updates, re-evaluate and produces maps; static and digital, to cater to the need for the most update stock-take of information that can be gained from various source of exploration output; seismic, full tensor gradiometry, assets; cables, pipelines, platforms, as well as discoverable resources. This year's them aims to include various prospects and initiative to align the industry itself with lower carbon emission and to explore the option for carbon capture storage (CCS) attempts in the popular basins such as the Malay and Penyu Basin. This is a big follow-up with the closing of MBR 2022 with the PSC signing for 9 blocks a few days earlier.

Credit: Sh Shahira Wafa Syed Khairulmunir Wafa

Over ~70 maps for unique blocks have been produced during the finalization stage, ~210 maps during data evaluation and additional 20 for the event. And this excludes the standardized maps to formalize information requested by prospective bidders as well as clients who are facing prospects of extending their contract.

The standardization of the map requires the optimization of workflow and standard templates to cater to rapid changes and exporting to rapid output.

For more information on the event, please access the following resources:

PETRONAS: Malaysia Bid Round

PETRONAS myPROdata

The Malaysian Reserve: Petronas offers 10 exploration blocks in MBR 2023

zero to pandas

There are alot of Python courses out there that we can jump into and get started with. But to a certain extent in that attempt to learn the language, the process becomes unbearably long and frustratingly slow. We all know the feeling of wanting to run before we could learn how to walk; we really wanna get started with some subtantial project but we do not know enough to even call the data into the terminal for viewing.

Back in August, freeCodeCamp in collaboration with Jovian.ai, organized a very interesting 6-week MOOC called Data Analysis with Python: Zero to Pandas and as a self-proclaimed Python groupie, I pledged my allegiance!

If there are any expectation that I've managed to whizz myself through the course and obtained a certificate, nothing of that sort happened; I missed the deadline cause I was busy testing out every single code I found and work had my brain on overdrive. I can't...I just...can't. Even with the extension, I was short of 2 Pythonic answers required to earn the certificate. But don't mistake my blunders for the quality of the content this course has to offer; is worth every gratitude of its graduates!

Zero to Pandas MOOC is a course that spans over 6 weeks with one lecture webinar per week that compacts the basics of Python modules that are relevant in executing data analysis. Like the play on its name, this course assumes no prior knowledge in Python language and aims to teach prospective students the basics on Python language structure AND the steps in analyzing real data. The course does not pretend that data analytics is easy and cut-corners to simplify anything. It is a very 'honest' demonstration that effectively gives overly ambitious future data analysts a flick on the forehead about data analysis. Who are we kidding? Data analysis using programming language requires sturdy knowledge in some nifty codes clean, splice and feature engineer the raw data and real critical thinking on figuring out 'Pythonic' ways to answer analytical questions. What does it even mean by Pythonic ways? Please refer to this article by Robert Clark, How to be Pythonic and Why You Should Care. We can discuss it somewhere down the line, when I am more experienced to understand it better. But for now, Packt Hub has the more comprehensive simple answer; it simply is an adjective coined to describe a way/code/structure of a code that utilizes or take advantage of the Python idioms well and displays the natural fluency in the language.

The bottom line is, we want to be able to fully utilize Python in its context and using its idioms to analyze data.

The course is conducted at Jovian.ai platform by its founder; Aakash and it takes advantage of Jupyter-like notebook format; Binder, in addition to making the synchronization available at Kaggle and Google's Colab. Each webinar in this course spans over close to 2 hours and each week, there are assignments on the lecture given. The assignments are due in a week but given the very disproportionate ratio of students and instructors, there were some extensions on the submission dates that I truly was grateful for. Forum for students is available at Jovian to engage students into discussing their ideas and question and the teaching body also conducts office hours where students can actively ask questions.

The instructor's method of teaching is something I believe to be effective for technical learners. In each lectures, he will be teaching the codes and module requires to execute certain tasks in the thorough procedure of the data analysis task itself. From importing the .csv formatted data into Python to establishing navigation to the data repository...from explaining what the hell loops are to touching base with creating functions. All in the controlled context of two most important module for the real objective of this course; Numpy and Pandas.

My gain from this course is immensely vast and that's why I truly think that freeCodeCamp and Jovian.ai really put the word 'tea' to 'teachers'. Taking advantage of the fact that people are involuntarily quarantined in their house, this course is something that should not be placed aside in the 'LATER' basket. I managed to clear my head to understand what 'loop' is! So I do think it can solve the world's problem!

In conclusion, this is the best course I have ever completed (90%!) on data analysis using Python. I look forward to attending it again and really finish up that last coursework.

Oh. Did I not mention why I got stuck? It was the last coursework. We are required to demonstrate all the steps of data analysis on data of our choice, create 5 questions and answer them using what we've learned throughout the course. Easy eh? Well, I've always had the tendency of digging my own grave everytime I get awesome cool assignments. But I'm not saying I did not do it :). Have a look-see at this notebook and consider the possibilities you can grasp after you've completed the course. And that's just my work...I'm a standard C-grade student.

And the exciting latest news from Jovian.ai is that they have upcoming course at Jovian for Deep Learning called Deep Learning with PyTorch: Zero to GANS! That's actually yesterday's news since they organized it earlier this year...so yeah...this is an impending second cohort! Tentatively, the course will start on Nov 14th. Click the link below to sign-up and get ready to attack the nitty-gritty. Don't say I didn't warn ya.

And that's me, reporting live from the confinement of COVID pandemic somewhere in a developing country at Southeast Asia....

With this, I am commencing my submission for the #30DayMapChallenge for 2023 🗺

The categories outlined is similar to that of last year but I am never going to hate this repetition. How can I? It's a basics of making maps and there's so much to learn from the single-word theme.

Any aspiring map-makers out there? Let's share our maps for this wonderful month of November under the #30DayMapChallenge 2023!

📚 Nature in the Heart of Borneo (2020)

Tool: ArcGIS Pro 2.6.1

Technique: Annotation, Labeling and Symbology

A series of maps were created for the book published by WWF-Malaysia and FORMADAT (Forum Masyarakat Adat Dataran Tinggi Borneo) back in 2020 called Nature in the Heart of Borneo.

This book was meant as a guide to some of the natural attractions at Northern parts of Sarawak. If it was clear, Northern Sarawak is where the we have our very own highlanders which consist of primarily the Lundayeh/Lun Bawang, Sa'ban and Kelabit people. Some of the beautiful settlements up in the north that should not be missed are Ba'kelalan and Long Semadoh. They have beautiful homestays and even more beautiful landscapes with trekking activities lined up for tourists. And this is the culmination of ardent passion by my two absolutely wonderful colleagues, Alicia Ng and Cynthia Chin.

Most part of the maps were made using readily available basemap provided by Esri in their Living Atlas. But in entirety, many of the features and details are drawn manually within ArcGIS Pro. Like many other mapmakers out there, the labeling feature is horrendously temperamental and I either end up using annotations instead.

In summary, technically, there are 2 lessons learned here:

1️⃣ Establish concept or pick an idea before you start drawing

A concept of the map and palette should be established at the earliest stage possible. And don't just throw the task of making maps and split them evenly between cartographers. They won't have similar ideas or similar interpretations of the concept. It'll only give you double the pain of creating the maps again from scratch.

2️⃣ Omit borders

If you're making maps for books, don't border trying to make borders and fully utilize the whole layout. In the end, you'll need to export out your maps and they will resize it anyway and it'll compromise the maps you created. As if it wasn't graining enough in the first place, it'll look absolutely microscopic by the time they're done.

Floating...in the mid of it

Please come back home