This should prove to be an interesting series of posts coming up, as I am working on a new project that is very unique and interesting. The idea is to use incoming data from Arduinos, Raspberry Pis, Gallileos, Edisons and other assortments of IoT type devices connected to oil and gas pipelines to determine if a leak is currently in progress and also predict if a leak is likely to occur in the future based on current and trending conditions.
My part in the project is all back end analytics, and I have very little to do with the actual telemetry and hardware. The telemetry will be posted using Azure Event Hubs, and thus my portion of the project begins with mocking that real time data at a large enough geo dispersed scale that I can develop a system that can handle it, and then switch my configurations to consume from the production event hubs. Since I am no longer a consultant working on projects with trade secrets and everything these days is about the elevation of skills in the community, I have posted everything on github that you can download and peruse at your leisure. Please note that this is in progress and well the github source may not necessarily work when you look. I’ll try to enforce a standard to comment “working – comment” on pushes to the repository. The git repository is located here: https://github.com/drcrook1/OilGas
This article is one of those that is going to help remind me how to do this deployment, as it can be a bit tricky. If you are working with F# for web jobs, like I have started doing, there are a few steps.
Create a new console application
Add proper nuget packages
Manually add a .dll reference and copy said .dll to output
So you are going to notice a slight shift in this blog to start incorporating not only video game development, but hardcore data analytics. As part of that shift, I am going to start incorporating F# into my standard set of languages as it is the language of hardcore data analytics if you roll with the .NET stack.
This particular article is about building a console based blob manager in F# instead of C#. The very first thing I noticed about using F# to manage my blobs as opposed to C# is just the sheer reduction in lines of code. The code presented here is a port of the C# article located here. This code will eventually make its way into a production system which is part of a big data solution I am building. New data sets that we acquire will be uploaded into blob storage, an entry stored into a queue, with a link to the data set. Once a job is prepared to run, the data will be moved to Hadoop to do the processing and then stored in its final location. So step 1 is…Store data in Blob storage.
Welcome to Part 2! We will be discussing Binary Classification. So I hope many of you have started using AzureML. If not, you should definitely check it out. Here is the link to the dev center for it. This article series will focus on a few key points.
Understanding the Evaluation of each Model Type.
Understanding the published Web Service of each Model
If you are looking for how to build a simple how to get started, check out this article.
The series will be broken down into a three parts.