Sunday, December 2, 2018

Anomaly detection system POC

Got an opportunity to work on time series data and Anomaly detection application on Data center.

This system is capable of monitoring health of servers and provides remediation whenever server is unhealthy. Our Intelligent system uses Machine Learning to detect or forecast Anomalies and warn the Service providers to ensure timely action

· We have designed the Orchestrator in such a way that it monitors those specific services which are added to our database. When any service goes down it raises a ticket, then it will run corresponding remediation solution to resolve the issue. Finally it closes the ticket after it completes the resolution.

· ELK Stack is used to collect the data/stats from specific server for monitoring health, in turn we use machine learning models to predict the anomaly in the data captured for the specific duration.

· In the similar way our machine learning model also forecasts the future potential anomalies based on specified criteria and raise a warning ticket for timely action hence it is capable of preventing upcoming issues
Used Plotly for visualizing the plots.
Below are some of the important screenshots

Thursday, September 20, 2018

Image Kernels

Find interesting link on CNN

Reference link:

Keras Inception V3 weights loading error

Got below error while loading the weights

After i spent lot of time on google, i thought this is our network issue. I restarted my notebook and attempted. That's it.

Wednesday, September 19, 2018

Elevated permissions in SharePoint Designer 2013

In SPD 2010 we have option called "Replace permissions".
In SPD 2013 we have option with "App step". For this needs to do few configuration steps.

Reference Link:

Tuesday, September 18, 2018

Image segmentation with Intersection Over Union (IOU)

If we plot the data it looks like the below

Encoder and Decoder is the common Convolution model for Image segmentation
How Do We Evaluate Semantic Segmentation Models? 
  • Intersection Over Union (IOU)
  • IOU is a robust measure of segmentation accuracy

Plot predictions vs truth
num_samples = 20
fig = plt.figure(figsize=(10, 2*num_samples))

for i in range(0, 4*num_samples, 4):
    segment_pred = model.predict(np.array([X_test[i,...]]))
    ax = fig.add_subplot(num_samples, 4, i+1, xticks=[], yticks=[])
    segment_truth = y_test[i,:].reshape(im_size, im_size, 2)[:,:,1]
    ax.imshow(X_test[i,:].reshape(im_size, im_size, 3), interpolation='nearest')
    ax = fig.add_subplot(num_samples, 4, i+2, xticks=[], yticks=[])
    ax.imshow(segment_truth, interpolation='nearest')
    ax = fig.add_subplot(num_samples, 4, i+3, xticks=[], yticks=[])
    ax.imshow(segment_pred.reshape(im_size, im_size, 2)[:,:,1], interpolation='nearest')
    ax = fig.add_subplot(num_samples, 4, i+4, xticks=[], yticks=[])
    binary_pred = segment_pred.reshape(im_size, im_size, 2)[:,:,1] > 0.5
    ax.imshow(binary_pred, interpolation='nearest')

    binary_pred_flat = binary_pred.reshape(im_size* im_size)
    segment_truth_flat = segment_truth.reshape(im_size* im_size)
    intersection = np.sum((binary_pred_flat + segment_truth_flat == 2.0))
    union = np.sum(np.clip((binary_pred_flat + segment_truth_flat), 0.0, 1.0))

    iou = intersection/union
    ax.set_title('IOU = {}'.format(iou))

1st column- Original flower
2nd column --Target/Ground truth
3rd column represents Our network predicted
4th column Threshold prediction with IOU

Reference link:

Monday, September 17, 2018

Keras: Error while training the model

While running a Keras model have encountered an error like the below
Keras expects y-data in (N, 17) shape, not (N,) as have probably provided, that's why it raises an error.
class_index_one_hot = keras.utils.to_categorical(class_index, 17)
Output Ex: class_index_one_hot: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
y_train shape: (911, 17)
y_test shape: (449, 17)