VSCode Python: Refer global site packages inside virtual environment

Using virtual environments in Visual Studio Code is a great practise while working on python. However, I had noticed that whenever I create a virtual environment, I had to reinstall the commonly used packages like numpy again in the virtual environment folder. The typical size of all such packages for a python program using Keras packages is nearly 500 MB.

Though this is a very good - and probably essential - practise for production environment, I did not want to use up a lot of unnecessary disk place in my study laptop's hard disk. After seraching on internet I could get some solutions to this problem like setting pythonPath variable in settings.json file under the project folder.

But I was curious and wanted to look for other possible options, and finally I found a setting in the pyenv.cfg file under the virtual environment folder. This file has a setting include-system-site-packages. Setting it to true results in the virtual environment automatically refering to global package library folder, like so

home = C:\Program Files\Python36

include-system-site-packages = true

version = 3.6.8

SharePoint JSOM queries

Accessing SharePoint through JSOM is a very flexible and intuitive way of working with SharePoint.

Its advantages are:

1. Data can be accessed from any browser, in any application.
2. Javascript can be modified easily, and hence shorter deployment cycles.
3. Can be easily embedded in SharePoint hosted apps or even a content query webpart, providing vital ability to customize sharepoint OOB elements (such as menus, styles, links, etc.) without a lot of overhead of development, testing and deployment.

To get started with JSOM the following articles are a great read.

https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/complete-basic-operations-using-javascript-library-code-in-sharepoint

https://www.sharepointsky.com/jsom-sharepoint/

https://www.codeproject.com/Articles/841321/How-to-initialize-the-SharePoint-Client-Context-in

https://docs.microsoft.com/en-us/sharepoint/dev/spfx/web-parts/guidance/connect-to-sharepoint-using-jsom

SharePoint framework webpart - ERR_SSL_PROTOCOL_ERROR

I was unable to view a newly created sharepoint framework webpart in Chrome browser and the error shown was

This site can’t provide a secure connection
localhost sent an invalid response.
Try running Windows Network Diagnostics.
ERR_SSL_PROTOCOL_ERROR

The only solution I could find for this problem was to change protocol of the local website from https to http.

Specifially, open serve.json file under config directory and set https = false, and initialPage = http://localhost:5432/workbench

{

  "$schema": "https://developer.microsoft.com/json-schemas/core-build/serve.schema.json",

  "port": 4321,

  "https": false,

  "initialPage": "http://localhost:5432/workbench",

  "api": {

    "port": 5432,

    "entryPath": "node_modules/@microsoft/sp-webpart-workbench/lib/api/"

  }

}

If you know of any other way to fix this error without changing protocol, please do let me know.

More information can be found on this blog https://github.com/SharePoint/sp-dev-docs/issues/2343

Configure MVC app to receive and handle OPTIONS requests

Many of the frequently used http request methods are readily available from within MVC controllers. e.g, GET, PUT, POST, DELETE, etc.

However, there is also one important HTTP request method OPTIONS, which is sent before any CORS request to check if the request is allowed. This method is normally handled automatically by MVC framework, but if we ever need to handle it inside our controller, then add following tags to web.config


<system.webServer>

  <handlers>

    <remove name="ExtensionlessUrlHandler-Integrated-4.0" />

    <remove name="OPTIONSVerbHandler" />

    <add name="ExtensionlessUrlHandler-Integrated-4.0" path="*." verb="*" type="System.Web.Handlers.TransferRequestHandler" preCondition="integratedMode,runtimeVersionv4.0" />

  </handlers>

</system.webServer>

This is explained in detail at https://docs.microsoft.com/en-us/aspnet/web-api/overview/security/enabling-cross-origin-requests-in-web-api

Azure Pipelines Error: Could not load file or assembly 'System.IO.FileSystem, Version=4.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

An angular project deployed through Azure DevOps pipeline would throw this error in the release pipeline:

 ##[error]System.Management.Automation.MethodInvocationException: Exception calling "Transform" with "4" argument(s): "Could not load file or assembly 'System.IO.FileSystem, Version=4.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified." ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.IO.FileSystem, Version=4.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

The error was thrown in a 'Config transformation' task. I tried to install file system package using

Install System.IO.FileSystem -Version 4.0.1

in a Powershell task and also tried to install dot net framework 4.6 using

Install DotNetFramework.Runtime -Version 4.6.1

in a custom Nuget task, but both these methods didn't solve the issue.

Finally after going through the task history realized that the task was auto updated to latest version which has caused this issue.

So simply changing this task's version from 2.* to 1.* solved this problem. This behavior is kind of expected since a major version includes some breaking changes by convention.

MVC unobstrusive - validate only a single form control

I was trying to find articles detailing how to validate only a single form control using unobstrusive validation, but could not find one. So I digged in a little and found out that the following way works nicely. It may not be the most ideal solution for this problem though, so your suggestions are welcome.

var myForm = $('form');
var validator = myForm.data('validator');
var myValidator = new $.validator(validator.settings, myForm);
myValidator.check($('#myElement')[0]);
myValidator.showErrors();

The key part here is to call check method on a validator object, this method is an internal method so is not available on the default validator object. So I have created a new validator object and passed settings of existing validator objects to it. It is important to pass existing settings since they contain final validation message for each element after combining jquery default message and any custom messages from server side attributes.

MVC validate a model using data annotation attributes without model binding

Sometimes, we need to validate a data model without model binding. For example, if the application has separate models for the actual data entity and an action method such as edit action in a view. In such cases, it would be very convenient to use data annotation attributes already specified on the model to perform validation logic.


This can be achieved by using ValidationContext and Validator objects, like so:

 var widget = new Widget
                        {
                            Id = 12,
                            Price = 15.57M
                        };

        var context = new ValidationContext(widget, null, null);
        var results = new List<ValidationResult>();
        if( Validator.TryValidateObject( widget, context, results, true ) )
        {
            //Validation Successful
        }
        else
        {
            //Validation Failed
        }

More information can be found at https://stackoverflow.com/questions/7101800/can-i-check-modelstate-without-modelbinding

Preventing overposting attacks in MVC

An mvc website can have seriously compromised security if it is not carefully defended against overposting attacks. Attackers can over post request data if controller actions use some sort model binding.

More information about these attacks and how to defend against them can be found at

https://andrewlock.net/preventing-mass-assignment-or-over-posting-in-asp-net-core/

c# Custom List validation attribute

By default, MVC model binder validates each object before assigning properties, including each object in a list.

However if we want to implement a custom validation attribute (using DataAnnotations), and use  Validator.TryValidateObject mehtod, this can get bit tricky, since it does not recursively validate 'properties of objects returned by properties'.

What is means it assuming I have class A defined as:

        class A

        {

            [Required]

            public string name;

        }

And class B defined as:

        class B

        {

            [Required]

            public List<A> myPropertyList;

        }

Then calling Validator.TryValidateObject on an object of will validate if myPropertyList inside the object has a value, but it will not validate if each object in the myPropertyList has a name assigned.



There are several solutions to this problem.

1) For an IEnumarable use custom validation attribute IsValid method convert myPropertyList to a IEnumarable  then enumerate through each object in the list and call Validator.TryValidateObject method on the object, finally combine all the result to get the error messages. This is explained at https://stackoverflow.com/questions/51523346/validating-lists-in-c-sharp


2) TryValidateObject method would also ignore validation attribute of name in class A if it is not a list, but a single object


e.g. given that I have

        class C

        {

            [Required]

            public string name;

        }

        class D

        {

            [Required]

            public A myProperty;

        }

e.g. given that I have e.g. given that I have e.g. given that I have e.g. given that I have 

TryValidateObject again won't validate that name is not empty if called on an object of class D. In such a situation a custom validation attribute can be assigned to myProperty which can do this task and aggregate the results. This is explained at http://www.technofattie.com/2011/10/05/recursive-validation-using-dataannotations.html

3) Implement IValidatableObject interface at the top level class and iterate through each property recursively to validate it.

4) There is also in interesting article for client side validation at https://stackoverflow.com/questions/23337170/data-validation-for-every-item-in-a-list-of-my-viewmodel

Powershell - Organize media files based on creation date

In windows media files like jpg, mp4 or mov have properties called as extension attributes which record their creation date in addition to the regular creation and modified date properties. When we copy of move files to some other location, the regular created date and modified date properties are changed, but not so with the extension attributes. Thus the extension attributes record  correct timestamps.

I found a handy powershell script which can be used to organise media files based on their extension attributes. This script can be found at http://todd.ropog.com/organize-media-files-with-powershell/

Saves us a lot of time while taking backups :)

Unit test controller actions with automapper

When testing controller actions we frequently come across objects which are converted using AutoMapper. While this library is very convenient to use, some extra coding is required in unit tests to test such controller actions. The following article provides a solution for this scenario:

https://stackoverflow.com/questions/49934707/automapper-in-xunit-testing-and-net-core-2-0

In practice, I added a custom mapping profile like so:

//auto mapper configuration
            var mockMapper = new MapperConfiguration(cfg =>
            {
                cfg.AddProfile(new MyMapperProfile());
            });
            var mapper = mockMapper.CreateMapper();

SharePoint Provider Hosted App: AuthenticationContext.HttpContext.User is Null after SharePointContextProvider.CheckRedirectionStatus

The issue was AuthenticationContext.HttpContext.User object is Null after SharePointContextProvider.CheckRedirectionStatus call inside OnAuthentication() method of custom ActionFilterAttribute of a provider hosted app.

public void OnAuthentication(AuthenticationContext filterContext)
        {
            if (filterContext == null)
            {
                throw new ArgumentNullException("filterContext");
            }
            bool ok = false;
            Uri redirectUrl;
            switch (SharePointContextProvider.CheckRedirectionStatus(filterContext.HttpContext, out redirectUrl))
            {
                case RedirectionStatus.Ok:
                    ok = true;
                    break;
                case RedirectionStatus.ShouldRedirect:
                    filterContext.Result = new RedirectResult(redirectUrl.AbsoluteUri);
                    break;
                case RedirectionStatus.CanNotRedirect:
                    filterContext.Result = new ViewResult { ViewName = "Error" };
                    break;
            }
            if (ok)
            {
 
                filterContext.Principal = filterContext.HttpContext.User;

Reason for this error was that the app was not registered in the dev server. After installing the app using AppRegNew.aspx this error was solved.

Sharepoint 2013 MVC 5 provider-hosted app. Fails to authenticate using SharePointContextProvider.CheckRedirectionStatus

One of the reasons for this failure is SPHostUrl parameter has not been passed to the post action. It can be passed by several ways

1. Use Html.BeginForm like so:

ViewBag.SPURL = Request.QueryString["SPHostUrl"];

And if the view

@using (Html.BeginForm("Upload", "Home", new { SPHostUrl = @ViewBag.SPURL }, FormMethod.Post, new { enctype = "multipart/form-data" }))

2. In C# pass the parameter in redirect statement

return RedirectToAction("Index", new {SPHostUrl = SharePointContext.GetSPHostUrl(HttpContext.Request).AbsoluteUri});

References:

https://social.technet.microsoft.com/Forums/office/en-US/d3d70290-520a-4975-848e-fd55e770b42f/sharepoint-2013-mvc-providerhosted-app-fails-to-pass-sharepointcontextfilter-on-httppost?forum=sharepointgeneral

https://codeday.me/en/qa/20190312/15428.html

SharePoint Provider Hosted App: SPAppWebUrl missing

I faced this problem in one of the sharepoint apps I was working on, a quick search on google returned few pages which suggest a solution:

Add a dummy list to the app: This solution works since sharepoint automatically creates app web for the app if there is some content to be deployed to the web.

More references:

https://stackoverflow.com/questions/18334525/cant-get-spappweburl-sharepoint-2013

https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/url-strings-and-tokens-in-sharepoint-add-ins?redirectedfrom=MSDN#Anchor_0

https://social.msdn.microsoft.com/Forums/en-US/7c417bb9-4456-4f67-a71b-97928369e040/how-to-add-standardtokens-to-aspx-page?forum=appsforsharepoint

https://sharepoint.stackexchange.com/questions/104063/sharepoint-app-spappweburl-query-string-is-missing-after-using-sharepointcont

Linq predicate builder

A predicate builder can be used to dynamically add conditions to a linq query.

The class is

using System;
using System.Linq;
using System.Linq.Expressions;
using System.Collections.Generic;
 
public static class PredicateBuilder
{
  public static Expression<Func<T, bool>> True<T> ()  { return f => true;  }
  public static Expression<Func<T, bool>> False<T> () { return f => false; }
 
  public static Expression<Func<T, bool>> Or<T> (this Expression<Func<T, bool>> expr1,
                                                      Expression<Func<T, bool>> expr2)
  {
    var invokedExpr = Expression.Invoke (expr2, expr1.Parameters.Cast<Expression> ());
    return Expression.Lambda<Func<T, bool>>
          (Expression.OrElse (expr1.Body, invokedExpr), expr1.Parameters);
  }
 
  public static Expression<Func<T, bool>> And<T> (this Expression<Func<T, bool>> expr1,
                                                       Expression<Func<T, bool>> expr2)
  {
    var invokedExpr = Expression.Invoke (expr2, expr1.Parameters.Cast<Expression> ());
    return Expression.Lambda<Func<T, bool>>
          (Expression.AndAlso (expr1.Body, invokedExpr), expr1.Parameters);
  }
}

And it can be used like:

var predicate = PredicateBuilder.False<Product>();

  foreach (string keyword in keywords)
  {
    string temp = keyword;
    predicate = predicate.Or (p => p.Description.Contains (temp));
  }

The complete source code can be found at http://www.albahari.com/nutshell/predicatebuilder.aspx 

Machine Learning Algorithms

We can find some interesting and informational e-books on the matlab web site at

https://uk.mathworks.com/campaigns/offers/machine-learning-with-matlab.confirmation.html?elqsid=1564492503463&potential_use=Home


Some interesting algorithms in Machine Learning
https://www.kdnuggets.com/2016/08/10-algorithms-machine-learning-engineers.html

Regex to find a string NOT followed by another

Visual studio has slightly different regex syntax. Thus the not symbol is ?! instead of ^ in normal regex.


Lets say we want to find all occurrences of StrA NOT followed by StrB and replace them with StrC

that is given the following data


StrA
StrB
StrAStrB
StrAXYZ
XYZStrA
StrBXYZ

We would like out output to be

StrC         //replaced  StrA since it is not being followed by StrB
StrB
StrAStrB
StrCXYZ      //replaced  StrA since it is not being followed by StrB
XYZStrC      //replaced  StrA since it is not being followed by StrB
StrBXYZ

The regular expression for find is: \b([a-zA-Z0-9_]*)StrA(?!StrB)([a-zA-Z0-9_]*)\b
And for replace is:                          $1StrC$2

TSQL: Convert string in mm:ss.ms format to time

The following statement converts time string (nvarchar) in mm:ss.ms format to Time

print DATEADD(ms, (ISNULL(LEFT(TRY_PARSE(RIGHT(ISNULL(LEFT(RIGHT(@timeString,LEN(@timeString)-CHARINDEX(':',@timeString)), 5), '0'),LEN(ISNULL(LEFT(RIGHT(@timeString,LEN(@timeString)-CHARINDEX(':',@timeString)), 5), '0'))-CHARINDEX('.',ISNULL(LEFT(RIGHT(@timeString,LEN(@timeString)-CHARINDEX(':',@timeString)), 5), '0'))) AS INT), 2), 0)) * 10,
DATEADD(s, ISNULL(TRY_PARSE(LEFT(ISNULL(LEFT(RIGHT(@timeString,LEN(@timeString)-CHARINDEX(':',@timeString)), 5), '0'),LEN(ISNULL(LEFT(RIGHT(@timeString,LEN(@timeString)-CHARINDEX(':',@timeString)), 5), '0'))-CHARINDEX('.',ISNULL(LEFT(RIGHT(@timeString,LEN(@timeString)-CHARINDEX(':',@timeString)), 5), '0'))) AS INT), 0),
DATEADD(mi, ISNULL(TRY_PARSE(LEFT(@timeString,CHARINDEX(':',@timeString)-1) AS INT), 0), '0:0')))

DACPAC deployment error: The database containment option has been changed to None.

This error primarily occurs for Azure SQL databases if we are trying to deploy dacpac package using SqlPackage.exe. It can be resolved by adding the following tag under <PropertyGroup></PropertyGroup>

<Containment>Partial</Containment>

List all versions of a nuget package

Use these commands in package manager console to list all versions of a Nuget package

$x = Find-Package UkSport.UKSportHub.Auth2 -AllVersions -ExactMatch
$x.Versions | Format-Table

Publish provider hosted app

The following are steps to publish a provider hosted app in brief:

1.  Register the app in the site to which it will be added using AppRegNew.aspx
2. From Visual Studio (VS) generate the app manifest file for the SharePoint Addin Project (not the Remote Web project) by publishing the app to a package file.
3.  Publish the Remote Web Project to the IIS server
4.  Add the app to the app catalouge
5. Add the app to the site collection

We can also view apps using AppPrincipals.aspx and grant them permissions using AppInv.aspx

SQL Server export/import database diagrams

Database diagrams are saved in sysdiagrams system table, hence they cannot be exported or imported directly. However the following steps are one of the alternatives to export and then import them, provided none of the tables involved the diagrams have changed in any way:

These steps assume that name of the database from which we want to export diagrams is 'MyDB' located at 'MyServer' sql instance

1. Export sysdiagrams table to a temporary sql server database
• Create a temporary database on the same sql server instance, say 'TempDB'
• Right click on 'MyDB'
• Select Tasks -> Export Data
• In the export data wizard input following values:
• Choose a Data Source
• Data Source: SQL Server Native Client
• Server Name: MyServer
• Authentication: Use applicable details
• Database: MyDB
• Choose a Destination
• Data Source: SQL Server Native Client
• Server Name: MyServer
• Authentication: Use applicable details
• Database: TempDB
• Specify Table copy or query: Write a query
• Provide a source query: select * from sysdiagrams 
• Map the query to a table name at destination such as 'TempTable'
• Run the query.
• The above steps copy data from sysdiagrams table in to a temporary database. While importing them back, use Generate scripts option to generate insert statements from the TempTable to a sql query. 
• Use these insert statements to import back the diagrams.

SQL Server Error: Could not obtain information about Windows NT group/user

I got this error while trying to create a new database diagram after restoring it from backup. The error indicates that current login does not have authorization necessary to get information about the database owner. This can be easily solved by executing following script

USE MyDB 
GO 
ALTER DATABASE MyDB set TRUSTWORTHY ON; 
GO 
EXEC dbo.sp_changedbowner @loginame = N'sa', @map = false 
GO 
sp_configure 'show advanced options', 1; 
GO 
RECONFIGURE; 
GO 
sp_configure 'clr enabled', 1; 
GO 
RECONFIGURE; 
GO

Copied from: https://stackoverflow.com/questions/25845836/could-not-obtain-information-about-windows-nt-group-user