Some t(r)ip with RabbitMQ

Recently, i made lots of experiments with RabbitMq with my friends and colleagues Alessandro Argiolas and Stefano Lazzati. We were interested in decoupling logic from ServiceNow to custom code, but, since we cannot leverage on powerful servers and we haven't permission to go to cloud, we were looking for a solution not to sugger of overflow.

We tried to use RabbitMQ, a lightweight message broker, as a workaround for this. The main idea is: if i have to ingest lot of data but i have low-performance hardware, can i do something? In our opinion, an option could be this one. If you don't want to install RabbitMQ you can use it as a service with CloudAMQP which is also fully documented, moreover there are tons of posts (written better than mine) that helps you to understand why is better to use a Message Broker and not a old-fashioned relational DB, what is an exhange, the routing modes and bla bla bla, so feel free to ask in the comments, but first read this.

So, about our architecture, consider those 4 application items:

1. Source System: just the pre-existing system that need to call the web service to do something
2. Ingestion Web Service: a really lightweight web service which just receive data and put it in the respective RabbitMQ queue. The message must be the most lightweight one
3. RabbitMQ Queue: a queue defined on RabbitMQ, will contain the messages the engine need to work
4. C# Engine: subscribed to the queue, as soon messages arrive ti work the case and return result to Source System and ACK to RabbitMQ. C# because we love C# but it could be whatever language
   

 

Ingestion Web Service 

Must be something easy peasy and lightweight: we written it in C# but maybe Go or other languages could be a better choice. This web services is your producer, it takes messages, it route message to the correct queue: in our scenario we realized queue for every customer, method and technology which is in the message request, so the routing key is generated from the info in the request.

The web service mainly contains a controller with this method

public class QueueController : ApiController
{
	private static readonly ILog _log = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);

	private readonly IQueueProducer _producer;
	private readonly string _exchange;

	public QueueController(IQueueProducer producer)
	{
		_producer = producer;

		using (SynopsContext context = new SynopsContext())
		{
			_exchange = context.GetCommonConfig("rabbit")["exchange"];
		}
	}

	[HttpPost]
	public IHttpActionResult AddToQueue(QueueRequest request)
	{
		string routingKey = string.Format("{0}_{1}_{2}", request.Customer.ToLower(), request.Method.ToLower(), request.Technology.ToLower());

		try
		{
			string jsonContent = JsonConvert.SerializeObject(request.Arguments);
			byte[] data = Encoding.Default.GetBytes(jsonContent);

			_producer.BasicPublish(_exchange, routingKey, data);
			_producer.Dispose();
		}
		catch (Exception ex)
		{
			_log.Error(ex.Message);

			return InternalServerError(ex);
		}

		return Ok();
	}
}

as you can see nothing so complex, the producer here 

public class RabbitMQProducer : IQueueProducer
{
	private readonly IConnection _connection;
	private readonly IModel _channel;

	public RabbitMQProducer(string hostname, string username, string password)
	{
		ConnectionFactory connectionFactory = new ConnectionFactory
		{
			HostName = hostname,
			UserName = username,
			Password = password
		};

		_connection = connectionFactory.CreateConnection();
		_channel = _connection.CreateModel();
	}

	public void BasicPublish(string exchange, string routingKey, byte[] data)
	{
		IBasicProperties properties = _channel.CreateBasicProperties();
		properties.Persistent = true;

		_channel.BasicPublish(exchange, routingKey, properties, data);
	}

	public void Dispose()
	{
		_channel?.Dispose();
		_channel?.Close();
		_connection?.Dispose();
		_connection?.Close();
	}
}

C# Engine

Besides what's in RabbitMQ, you need a consumer to consume message. Fun fact, you can balance the load adding more than just one Engine because you are centralizing the queue on one machine. It's interesting how using prefecth count in QoS you can choose how many messages you want to process simultaneously. Instead to launch the correct code to process message depending on the functionality, we just used a strategy pattern and we simply launch a task for every message we want to work.

public class RabbitMQConsumer
{
	private readonly ILog _log = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);

	private readonly IConnection _connection;
	private readonly IModel _channel;
	private readonly EventingBasicConsumer _consumer;

	public RabbitMQConsumer()
	{
		Dictionary rabbitConfig;

		using (SynopsContext context = new SynopsContext())
		{
			rabbitConfig = context.GetCommonConfig("rabbit");
		}

		rabbitConfig.Add("password", ConfigurationManager.AppSettings["rabbit_password"]);

		ConnectionFactory connectionFactory = new ConnectionFactory
		{
			HostName = rabbitConfig["hostname"],
			UserName = rabbitConfig["username"],
			Password = rabbitConfig["password"],
		};

		ushort prefetchCount = ushort.Parse(rabbitConfig["prefetch"]);

		_log.Info(string.Format("Number of parallel tasks for each queue: {0}", prefetchCount));

		_connection = connectionFactory.CreateConnection();
		_channel = _connection.CreateModel();
		_channel.BasicQos(0, prefetchCount, false);
		_consumer = new EventingBasicConsumer(_channel);
		_consumer.Received += BasicConsume;
	}

	public void BasicConsume(string queue, bool autoAck = false)
	{
		_channel.BasicConsume(queue, autoAck, _consumer);
	}

	private void BasicConsume(object sender, BasicDeliverEventArgs args)
	{
		Task.Run(() => BasicConsume(args.Exchange, args.RoutingKey, args.DeliveryTag, args.Body));
	}

	private void BasicConsume(string exchange, string routingKey, ulong deliveryTag, byte[] data)
	{
		try
		{
			Strategy strategy = new Strategy(routingKey);

			strategy.Run(data);
		}
		catch (Exception ex)
		{
			_log.Error(string.Format("Error during working exchange {0} with routing key {1} and delivery tag {2}. With exception:", exchange, routingKey, deliveryTag, ex));
		}
		finally
		{
			_channel.BasicAck(deliveryTag, false);
			_log.Info(string.Format("Ack {0}.", deliveryTag));
		}
	}
}

Lesson Learned

• Use this technique when you can leverage on async approach 
• Consider monitoring for RabbitMQ and Engine, it's a new piece of software
• Messages must be REALLY lightweight, otherwise you will need a load balancer for the web service... and so you
  

Read More

ServiceNow API and Attachment API

Lately i am using ServiceNow as a human-in-the-loop tool with RPA. Fortunally, not only as a front-end but RPA is becoming more and more the arm of the BPM workflow in the customer systems, and it's really fun to make it work together.

In past projects i noticed that ServiceNow guys used to send file to external services using a sync call: this is for sure fast, reliable but is not a good option when files, converted in base64, is bigger than 32MB.

This means that if you need to pass files bigger than 25-26MB you must change the approach from sync to async one, and here's where Attachment API helps you, enabling file transfer for bigger files. It's clear that async approach needs you to control what's happening on the other side (but we will see it on another post).

To help working also with the others OOO API i found this project on github that manages lots of calls.

Unfortunally they don't manage upload of files through AttachmentAPI nor so many ways to download files, so i decided to fork the project (you can find it here) and to submit my code to the authors (i don't even know if they accepted or refused it) but i will give you just a bunch of code lines i used to edit the lib so you can replicate and use it.

In ServiceNowClient.cs just added two methods, for upload and download as a stream

public async Task DownloadAttachmentAsyncAsStream(Attachment attachment, string outputPath, string filename = null, CancellationToken cancellationToken = default)
{
	var response = await _httpClient.GetAsync(attachment.DownloadLink, cancellationToken).ConfigureAwait(false);
	var stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false);
	return stream;
}

public async Task UploadAttachmentAsync(string filename, string tablename, string tablesysid, byte[] data)
{
	HttpContent genericFileContent = new ByteArrayContent(data);
	string mimeType = MimeMapping.GetMimeMapping(filename);
	genericFileContent.Headers.Add("Content-Type", mimeType);

	string requestUri = "api/now/v1/attachment/file?file_name=" + filename + "&table_name=" + tablename + "&table_sys_id=" + tablesysid;
	var response = await _httpClient.PostAsync(requestUri, genericFileContent).ConfigureAwait(false);
	if (response == null)
	{
		throw new Exception("Null response.");
	}

	if (!response.IsSuccessStatusCode)
	{
		throw new Exception($"Server error {response.StatusCode} ({(int)response.StatusCode}): {response.ReasonPhrase}.");
	}

	return response.IsSuccessStatusCode;
}

then i added MimeMapping (find it below) which is just a copy of the one on .NET that is not in the library, and that's quite everything... simple but (hope so) useful.

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;

namespace ServiceNow.Api
{
	public static class MimeMapping
	{
		private static Dictionary _mappingDictionary;

		static MimeMapping()
		{
			Init();
		}

		public static string GetMimeMapping(string fileName)
		{
			if (fileName == null)
			{
				throw new ArgumentNullException("fileName");
			}

			string extension = Path.GetExtension(fileName);
			return _mappingDictionary[extension];
		}

		
		private static void Init()
		{
			// This list was copied from the IIS7 configuration file located at:
			// %windir%\system32\inetsrv\config\applicationHost.config
			_mappingDictionary = new Dictionary();
			_mappingDictionary.Add(".323", "text/h323");
			_mappingDictionary.Add(".aaf", "application/octet-stream");
			_mappingDictionary.Add(".aca", "application/octet-stream");
			_mappingDictionary.Add(".accdb", "application/msaccess");
			_mappingDictionary.Add(".accde", "application/msaccess");
			_mappingDictionary.Add(".accdt", "application/msaccess");
			_mappingDictionary.Add(".acx", "application/internet-property-stream");
			_mappingDictionary.Add(".afm", "application/octet-stream");
			_mappingDictionary.Add(".ai", "application/postscript");
			_mappingDictionary.Add(".aif", "audio/x-aiff");
			_mappingDictionary.Add(".aifc", "audio/aiff");
			_mappingDictionary.Add(".aiff", "audio/aiff");
			_mappingDictionary.Add(".application", "application/x-ms-application");
			_mappingDictionary.Add(".art", "image/x-jg");
			_mappingDictionary.Add(".asd", "application/octet-stream");
			_mappingDictionary.Add(".asf", "video/x-ms-asf");
			_mappingDictionary.Add(".asi", "application/octet-stream");
			_mappingDictionary.Add(".asm", "text/plain");
			_mappingDictionary.Add(".asr", "video/x-ms-asf");
			_mappingDictionary.Add(".asx", "video/x-ms-asf");
			_mappingDictionary.Add(".atom", "application/atom+xml");
			_mappingDictionary.Add(".au", "audio/basic");
			_mappingDictionary.Add(".avi", "video/x-msvideo");
			_mappingDictionary.Add(".axs", "application/olescript");
			_mappingDictionary.Add(".bas", "text/plain");
			_mappingDictionary.Add(".bcpio", "application/x-bcpio");
			_mappingDictionary.Add(".bin", "application/octet-stream");
			_mappingDictionary.Add(".bmp", "image/bmp");
			_mappingDictionary.Add(".c", "text/plain");
			_mappingDictionary.Add(".cab", "application/octet-stream");
			_mappingDictionary.Add(".calx", "application/vnd.ms-office.calx");
			_mappingDictionary.Add(".cat", "application/vnd.ms-pki.seccat");
			_mappingDictionary.Add(".cdf", "application/x-cdf");
			_mappingDictionary.Add(".chm", "application/octet-stream");
			_mappingDictionary.Add(".class", "application/x-java-applet");
			_mappingDictionary.Add(".clp", "application/x-msclip");
			_mappingDictionary.Add(".cmx", "image/x-cmx");
			_mappingDictionary.Add(".cnf", "text/plain");
			_mappingDictionary.Add(".cod", "image/cis-cod");
			_mappingDictionary.Add(".cpio", "application/x-cpio");
			_mappingDictionary.Add(".cpp", "text/plain");
			_mappingDictionary.Add(".crd", "application/x-mscardfile");
			_mappingDictionary.Add(".crl", "application/pkix-crl");
			_mappingDictionary.Add(".crt", "application/x-x509-ca-cert");
			_mappingDictionary.Add(".csh", "application/x-csh");
			_mappingDictionary.Add(".css", "text/css");
			_mappingDictionary.Add(".csv", "application/octet-stream");
			_mappingDictionary.Add(".cur", "application/octet-stream");
			_mappingDictionary.Add(".dcr", "application/x-director");
			_mappingDictionary.Add(".deploy", "application/octet-stream");
			_mappingDictionary.Add(".der", "application/x-x509-ca-cert");
			_mappingDictionary.Add(".dib", "image/bmp");
			_mappingDictionary.Add(".dir", "application/x-director");
			_mappingDictionary.Add(".disco", "text/xml");
			_mappingDictionary.Add(".dll", "application/x-msdownload");
			_mappingDictionary.Add(".dll.config", "text/xml");
			_mappingDictionary.Add(".dlm", "text/dlm");
			_mappingDictionary.Add(".doc", "application/msword");
			_mappingDictionary.Add(".docm", "application/vnd.ms-word.document.macroEnabled.12");
			_mappingDictionary.Add(".docx", "application/vnd.openxmlformats-officedocument.wordprocessingml.document");
			_mappingDictionary.Add(".dot", "application/msword");
			_mappingDictionary.Add(".dotm", "application/vnd.ms-word.template.macroEnabled.12");
			_mappingDictionary.Add(".dotx", "application/vnd.openxmlformats-officedocument.wordprocessingml.template");
			_mappingDictionary.Add(".dsp", "application/octet-stream");
			_mappingDictionary.Add(".dtd", "text/xml");
			_mappingDictionary.Add(".dvi", "application/x-dvi");
			_mappingDictionary.Add(".dwf", "drawing/x-dwf");
			_mappingDictionary.Add(".dwp", "application/octet-stream");
			_mappingDictionary.Add(".dxr", "application/x-director");
			_mappingDictionary.Add(".eml", "message/rfc822");
			_mappingDictionary.Add(".emz", "application/octet-stream");
			_mappingDictionary.Add(".eot", "application/octet-stream");
			_mappingDictionary.Add(".eps", "application/postscript");
			_mappingDictionary.Add(".etx", "text/x-setext");
			_mappingDictionary.Add(".evy", "application/envoy");
			_mappingDictionary.Add(".exe", "application/octet-stream");
			_mappingDictionary.Add(".exe.config", "text/xml");
			_mappingDictionary.Add(".fdf", "application/vnd.fdf");
			_mappingDictionary.Add(".fif", "application/fractals");
			_mappingDictionary.Add(".fla", "application/octet-stream");
			_mappingDictionary.Add(".flr", "x-world/x-vrml");
			_mappingDictionary.Add(".flv", "video/x-flv");
			_mappingDictionary.Add(".gif", "image/gif");
			_mappingDictionary.Add(".gtar", "application/x-gtar");
			_mappingDictionary.Add(".gz", "application/x-gzip");
			_mappingDictionary.Add(".h", "text/plain");
			_mappingDictionary.Add(".hdf", "application/x-hdf");
			_mappingDictionary.Add(".hdml", "text/x-hdml");
			_mappingDictionary.Add(".hhc", "application/x-oleobject");
			_mappingDictionary.Add(".hhk", "application/octet-stream");
			_mappingDictionary.Add(".hhp", "application/octet-stream");
			_mappingDictionary.Add(".hlp", "application/winhlp");
			_mappingDictionary.Add(".hqx", "application/mac-binhex40");
			_mappingDictionary.Add(".hta", "application/hta");
			_mappingDictionary.Add(".htc", "text/x-component");
			_mappingDictionary.Add(".htm", "text/html");
			_mappingDictionary.Add(".html", "text/html");
			_mappingDictionary.Add(".htt", "text/webviewhtml");
			_mappingDictionary.Add(".hxt", "text/html");
			_mappingDictionary.Add(".ico", "image/x-icon");
			_mappingDictionary.Add(".ics", "application/octet-stream");
			_mappingDictionary.Add(".ief", "image/ief");
			_mappingDictionary.Add(".iii", "application/x-iphone");
			_mappingDictionary.Add(".inf", "application/octet-stream");
			_mappingDictionary.Add(".ins", "application/x-internet-signup");
			_mappingDictionary.Add(".isp", "application/x-internet-signup");
			_mappingDictionary.Add(".IVF", "video/x-ivf");
			_mappingDictionary.Add(".jar", "application/java-archive");
			_mappingDictionary.Add(".java", "application/octet-stream");
			_mappingDictionary.Add(".jck", "application/liquidmotion");
			_mappingDictionary.Add(".jcz", "application/liquidmotion");
			_mappingDictionary.Add(".jfif", "image/pjpeg");
			_mappingDictionary.Add(".jpb", "application/octet-stream");
			_mappingDictionary.Add(".jpe", "image/jpeg");
			_mappingDictionary.Add(".jpeg", "image/jpeg");
			_mappingDictionary.Add(".jpg", "image/jpeg");
			_mappingDictionary.Add(".js", "application/x-javascript");
			_mappingDictionary.Add(".jsx", "text/jscript");
			_mappingDictionary.Add(".latex", "application/x-latex");
			_mappingDictionary.Add(".lit", "application/x-ms-reader");
			_mappingDictionary.Add(".lpk", "application/octet-stream");
			_mappingDictionary.Add(".lsf", "video/x-la-asf");
			_mappingDictionary.Add(".lsx", "video/x-la-asf");
			_mappingDictionary.Add(".lzh", "application/octet-stream");
			_mappingDictionary.Add(".m13", "application/x-msmediaview");
			_mappingDictionary.Add(".m14", "application/x-msmediaview");
			_mappingDictionary.Add(".m1v", "video/mpeg");
			_mappingDictionary.Add(".m3u", "audio/x-mpegurl");
			_mappingDictionary.Add(".man", "application/x-troff-man");
			_mappingDictionary.Add(".manifest", "application/x-ms-manifest");
			_mappingDictionary.Add(".map", "text/plain");
			_mappingDictionary.Add(".mdb", "application/x-msaccess");
			_mappingDictionary.Add(".mdp", "application/octet-stream");
			_mappingDictionary.Add(".me", "application/x-troff-me");
			_mappingDictionary.Add(".mht", "message/rfc822");
			_mappingDictionary.Add(".mhtml", "message/rfc822");
			_mappingDictionary.Add(".mid", "audio/mid");
			_mappingDictionary.Add(".midi", "audio/mid");
			_mappingDictionary.Add(".mix", "application/octet-stream");
			_mappingDictionary.Add(".mmf", "application/x-smaf");
			_mappingDictionary.Add(".mno", "text/xml");
			_mappingDictionary.Add(".mny", "application/x-msmoney");
			_mappingDictionary.Add(".mov", "video/quicktime");
			_mappingDictionary.Add(".movie", "video/x-sgi-movie");
			_mappingDictionary.Add(".mp2", "video/mpeg");
			_mappingDictionary.Add(".mp3", "audio/mpeg");
			_mappingDictionary.Add(".mpa", "video/mpeg");
			_mappingDictionary.Add(".mpe", "video/mpeg");
			_mappingDictionary.Add(".mpeg", "video/mpeg");
			_mappingDictionary.Add(".mpg", "video/mpeg");
			_mappingDictionary.Add(".mpp", "application/vnd.ms-project");
			_mappingDictionary.Add(".mpv2", "video/mpeg");
			_mappingDictionary.Add(".ms", "application/x-troff-ms");
			_mappingDictionary.Add(".msi", "application/octet-stream");
			_mappingDictionary.Add(".mso", "application/octet-stream");
			_mappingDictionary.Add(".mvb", "application/x-msmediaview");
			_mappingDictionary.Add(".mvc", "application/x-miva-compiled");
			_mappingDictionary.Add(".nc", "application/x-netcdf");
			_mappingDictionary.Add(".nsc", "video/x-ms-asf");
			_mappingDictionary.Add(".nws", "message/rfc822");
			_mappingDictionary.Add(".ocx", "application/octet-stream");
			_mappingDictionary.Add(".oda", "application/oda");
			_mappingDictionary.Add(".odc", "text/x-ms-odc");
			_mappingDictionary.Add(".ods", "application/oleobject");
			_mappingDictionary.Add(".one", "application/onenote");
			_mappingDictionary.Add(".onea", "application/onenote");
			_mappingDictionary.Add(".onetoc", "application/onenote");
			_mappingDictionary.Add(".onetoc2", "application/onenote");
			_mappingDictionary.Add(".onetmp", "application/onenote");
			_mappingDictionary.Add(".onepkg", "application/onenote");
			_mappingDictionary.Add(".osdx", "application/opensearchdescription+xml");
			_mappingDictionary.Add(".p10", "application/pkcs10");
			_mappingDictionary.Add(".p12", "application/x-pkcs12");
			_mappingDictionary.Add(".p7b", "application/x-pkcs7-certificates");
			_mappingDictionary.Add(".p7c", "application/pkcs7-mime");
			_mappingDictionary.Add(".p7m", "application/pkcs7-mime");
			_mappingDictionary.Add(".p7r", "application/x-pkcs7-certreqresp");
			_mappingDictionary.Add(".p7s", "application/pkcs7-signature");
			_mappingDictionary.Add(".pbm", "image/x-portable-bitmap");
			_mappingDictionary.Add(".pcx", "application/octet-stream");
			_mappingDictionary.Add(".pcz", "application/octet-stream");
			_mappingDictionary.Add(".pdf", "application/pdf");
			_mappingDictionary.Add(".pfb", "application/octet-stream");
			_mappingDictionary.Add(".pfm", "application/octet-stream");
			_mappingDictionary.Add(".pfx", "application/x-pkcs12");
			_mappingDictionary.Add(".pgm", "image/x-portable-graymap");
			_mappingDictionary.Add(".pko", "application/vnd.ms-pki.pko");
			_mappingDictionary.Add(".pma", "application/x-perfmon");
			_mappingDictionary.Add(".pmc", "application/x-perfmon");
			_mappingDictionary.Add(".pml", "application/x-perfmon");
			_mappingDictionary.Add(".pmr", "application/x-perfmon");
			_mappingDictionary.Add(".pmw", "application/x-perfmon");
			_mappingDictionary.Add(".png", "image/png");
			_mappingDictionary.Add(".pnm", "image/x-portable-anymap");
			_mappingDictionary.Add(".pnz", "image/png");
			_mappingDictionary.Add(".pot", "application/vnd.ms-powerpoint");
			_mappingDictionary.Add(".potm", "application/vnd.ms-powerpoint.template.macroEnabled.12");
			_mappingDictionary.Add(".potx", "application/vnd.openxmlformats-officedocument.presentationml.template");
			_mappingDictionary.Add(".ppam", "application/vnd.ms-powerpoint.addin.macroEnabled.12");
			_mappingDictionary.Add(".ppm", "image/x-portable-pixmap");
			_mappingDictionary.Add(".pps", "application/vnd.ms-powerpoint");
			_mappingDictionary.Add(".ppsm", "application/vnd.ms-powerpoint.slideshow.macroEnabled.12");
			_mappingDictionary.Add(".ppsx", "application/vnd.openxmlformats-officedocument.presentationml.slideshow");
			_mappingDictionary.Add(".ppt", "application/vnd.ms-powerpoint");
			_mappingDictionary.Add(".pptm", "application/vnd.ms-powerpoint.presentation.macroEnabled.12");
			_mappingDictionary.Add(".pptx", "application/vnd.openxmlformats-officedocument.presentationml.presentation");
			_mappingDictionary.Add(".prf", "application/pics-rules");
			_mappingDictionary.Add(".prm", "application/octet-stream");
			_mappingDictionary.Add(".prx", "application/octet-stream");
			_mappingDictionary.Add(".ps", "application/postscript");
			_mappingDictionary.Add(".psd", "application/octet-stream");
			_mappingDictionary.Add(".psm", "application/octet-stream");
			_mappingDictionary.Add(".psp", "application/octet-stream");
			_mappingDictionary.Add(".pub", "application/x-mspublisher");
			_mappingDictionary.Add(".qt", "video/quicktime");
			_mappingDictionary.Add(".qtl", "application/x-quicktimeplayer");
			_mappingDictionary.Add(".qxd", "application/octet-stream");
			_mappingDictionary.Add(".ra", "audio/x-pn-realaudio");
			_mappingDictionary.Add(".ram", "audio/x-pn-realaudio");
			_mappingDictionary.Add(".rar", "application/octet-stream");
			_mappingDictionary.Add(".ras", "image/x-cmu-raster");
			_mappingDictionary.Add(".rf", "image/vnd.rn-realflash");
			_mappingDictionary.Add(".rgb", "image/x-rgb");
			_mappingDictionary.Add(".rm", "application/vnd.rn-realmedia");
			_mappingDictionary.Add(".rmi", "audio/mid");
			_mappingDictionary.Add(".roff", "application/x-troff");
			_mappingDictionary.Add(".rpm", "audio/x-pn-realaudio-plugin");
			_mappingDictionary.Add(".rtf", "application/rtf");
			_mappingDictionary.Add(".rtx", "text/richtext");
			_mappingDictionary.Add(".scd", "application/x-msschedule");
			_mappingDictionary.Add(".sct", "text/scriptlet");
			_mappingDictionary.Add(".sea", "application/octet-stream");
			_mappingDictionary.Add(".setpay", "application/set-payment-initiation");
			_mappingDictionary.Add(".setreg", "application/set-registration-initiation");
			_mappingDictionary.Add(".sgml", "text/sgml");
			_mappingDictionary.Add(".sh", "application/x-sh");
			_mappingDictionary.Add(".shar", "application/x-shar");
			_mappingDictionary.Add(".sit", "application/x-stuffit");
			_mappingDictionary.Add(".sldm", "application/vnd.ms-powerpoint.slide.macroEnabled.12");
			_mappingDictionary.Add(".sldx", "application/vnd.openxmlformats-officedocument.presentationml.slide");
			_mappingDictionary.Add(".smd", "audio/x-smd");
			_mappingDictionary.Add(".smi", "application/octet-stream");
			_mappingDictionary.Add(".smx", "audio/x-smd");
			_mappingDictionary.Add(".smz", "audio/x-smd");
			_mappingDictionary.Add(".snd", "audio/basic");
			_mappingDictionary.Add(".snp", "application/octet-stream");
			_mappingDictionary.Add(".spc", "application/x-pkcs7-certificates");
			_mappingDictionary.Add(".spl", "application/futuresplash");
			_mappingDictionary.Add(".src", "application/x-wais-source");
			_mappingDictionary.Add(".ssm", "application/streamingmedia");
			_mappingDictionary.Add(".sst", "application/vnd.ms-pki.certstore");
			_mappingDictionary.Add(".stl", "application/vnd.ms-pki.stl");
			_mappingDictionary.Add(".sv4cpio", "application/x-sv4cpio");
			_mappingDictionary.Add(".sv4crc", "application/x-sv4crc");
			_mappingDictionary.Add(".swf", "application/x-shockwave-flash");
			_mappingDictionary.Add(".t", "application/x-troff");
			_mappingDictionary.Add(".tar", "application/x-tar");
			_mappingDictionary.Add(".tcl", "application/x-tcl");
			_mappingDictionary.Add(".tex", "application/x-tex");
			_mappingDictionary.Add(".texi", "application/x-texinfo");
			_mappingDictionary.Add(".texinfo", "application/x-texinfo");
			_mappingDictionary.Add(".tgz", "application/x-compressed");
			_mappingDictionary.Add(".thmx", "application/vnd.ms-officetheme");
			_mappingDictionary.Add(".thn", "application/octet-stream");
			_mappingDictionary.Add(".tif", "image/tiff");
			_mappingDictionary.Add(".tiff", "image/tiff");
			_mappingDictionary.Add(".toc", "application/octet-stream");
			_mappingDictionary.Add(".tr", "application/x-troff");
			_mappingDictionary.Add(".trm", "application/x-msterminal");
			_mappingDictionary.Add(".tsv", "text/tab-separated-values");
			_mappingDictionary.Add(".ttf", "application/octet-stream");
			_mappingDictionary.Add(".txt", "text/plain");
			_mappingDictionary.Add(".u32", "application/octet-stream");
			_mappingDictionary.Add(".uls", "text/iuls");
			_mappingDictionary.Add(".ustar", "application/x-ustar");
			_mappingDictionary.Add(".vbs", "text/vbscript");
			_mappingDictionary.Add(".vcf", "text/x-vcard");
			_mappingDictionary.Add(".vcs", "text/plain");
			_mappingDictionary.Add(".vdx", "application/vnd.ms-visio.viewer");
			_mappingDictionary.Add(".vml", "text/xml");
			_mappingDictionary.Add(".vsd", "application/vnd.visio");
			_mappingDictionary.Add(".vss", "application/vnd.visio");
			_mappingDictionary.Add(".vst", "application/vnd.visio");
			_mappingDictionary.Add(".vsto", "application/x-ms-vsto");
			_mappingDictionary.Add(".vsw", "application/vnd.visio");
			_mappingDictionary.Add(".vsx", "application/vnd.visio");
			_mappingDictionary.Add(".vtx", "application/vnd.visio");
			_mappingDictionary.Add(".wav", "audio/wav");
			_mappingDictionary.Add(".wax", "audio/x-ms-wax");
			_mappingDictionary.Add(".wbmp", "image/vnd.wap.wbmp");
			_mappingDictionary.Add(".wcm", "application/vnd.ms-works");
			_mappingDictionary.Add(".wdb", "application/vnd.ms-works");
			_mappingDictionary.Add(".wks", "application/vnd.ms-works");
			_mappingDictionary.Add(".wm", "video/x-ms-wm");
			_mappingDictionary.Add(".wma", "audio/x-ms-wma");
			_mappingDictionary.Add(".wmd", "application/x-ms-wmd");
			_mappingDictionary.Add(".wmf", "application/x-msmetafile");
			_mappingDictionary.Add(".wml", "text/vnd.wap.wml");
			_mappingDictionary.Add(".wmlc", "application/vnd.wap.wmlc");
			_mappingDictionary.Add(".wmls", "text/vnd.wap.wmlscript");
			_mappingDictionary.Add(".wmlsc", "application/vnd.wap.wmlscriptc");
			_mappingDictionary.Add(".wmp", "video/x-ms-wmp");
			_mappingDictionary.Add(".wmv", "video/x-ms-wmv");
			_mappingDictionary.Add(".wmx", "video/x-ms-wmx");
			_mappingDictionary.Add(".wmz", "application/x-ms-wmz");
			_mappingDictionary.Add(".wps", "application/vnd.ms-works");
			_mappingDictionary.Add(".wri", "application/x-mswrite");
			_mappingDictionary.Add(".wrl", "x-world/x-vrml");
			_mappingDictionary.Add(".wrz", "x-world/x-vrml");
			_mappingDictionary.Add(".wsdl", "text/xml");
			_mappingDictionary.Add(".wvx", "video/x-ms-wvx");
			_mappingDictionary.Add(".x", "application/directx");
			_mappingDictionary.Add(".xaf", "x-world/x-vrml");
			_mappingDictionary.Add(".xaml", "application/xaml+xml");
			_mappingDictionary.Add(".xap", "application/x-silverlight-app");
			_mappingDictionary.Add(".xbap", "application/x-ms-xbap");
			_mappingDictionary.Add(".xbm", "image/x-xbitmap");
			_mappingDictionary.Add(".xdr", "text/plain");
			_mappingDictionary.Add(".xla", "application/vnd.ms-excel");
			_mappingDictionary.Add(".xlam", "application/vnd.ms-excel.addin.macroEnabled.12");
			_mappingDictionary.Add(".xlc", "application/vnd.ms-excel");
			_mappingDictionary.Add(".xlm", "application/vnd.ms-excel");
			_mappingDictionary.Add(".xls", "application/vnd.ms-excel");
			_mappingDictionary.Add(".xlsb", "application/vnd.ms-excel.sheet.binary.macroEnabled.12");
			_mappingDictionary.Add(".xlsm", "application/vnd.ms-excel.sheet.macroEnabled.12");
			_mappingDictionary.Add(".xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
			_mappingDictionary.Add(".xlt", "application/vnd.ms-excel");
			_mappingDictionary.Add(".xltm", "application/vnd.ms-excel.template.macroEnabled.12");
			_mappingDictionary.Add(".xltx", "application/vnd.openxmlformats-officedocument.spreadsheetml.template");
			_mappingDictionary.Add(".xlw", "application/vnd.ms-excel");
			_mappingDictionary.Add(".xml", "text/xml");
			_mappingDictionary.Add(".xof", "x-world/x-vrml");
			_mappingDictionary.Add(".xpm", "image/x-xpixmap");
			_mappingDictionary.Add(".xps", "application/vnd.ms-xpsdocument");
			_mappingDictionary.Add(".xsd", "text/xml");
			_mappingDictionary.Add(".xsf", "text/xml");
			_mappingDictionary.Add(".xsl", "text/xml");
			_mappingDictionary.Add(".xslt", "text/xml");
			_mappingDictionary.Add(".xsn", "application/octet-stream");
			_mappingDictionary.Add(".xtp", "application/octet-stream");
			_mappingDictionary.Add(".xwd", "image/x-xwindowdump");
			_mappingDictionary.Add(".z", "application/x-compress");
			_mappingDictionary.Add(".zip", "application/x-zip-compressed");
		}
	}
}

Read More

Tips'n'tricks installing AA11 Citrix Remote Agent

It's been a long time i want to write this little blog post. I had some experience with Automation Anywhere 11, trying to setup it in an environment i could not create any Virtual Machine: so the only solution was to try to automate via Citrix.

Since i'm not a big fan of surface automation, as you all, i found this plugin (which is really similar to the one also UiPath have onboard... instead BluePrism lacks of this feature). 

As you can see in the following picture that is taken from here it helps you to hosts the robot machine (so you are the service provider) without installing automation software on customers data center, and the commercial promise is to use Citrix Application as local application.

Customer must install only AARemoteAgent on Citrix Servers and you have to start the Citrix Receiver before starting yout robot and everything will work fine.

 

 

But you are here because of the lesson learned, not for the commercial tips, so what i've learned from this experience?

1. Latency: programming robot using Remote Agent could be frustrating because of the latency, consider overhead in development;
2. Functionality: as you can imagine, it works quite well but it doesn't works for everything. I tested it on some custom application and the result is: more custom application = more problems, somebody could argue that is the same with local application but it's also true that with local application you have more workaround available;
   
3. Internet explorer + Citrix: nobody wrote this but if Citrix server version is < 7.15, you will never been able to use IE through Citrix RemoteAgent;
   
4. Chrome + Citrix:sometimes it doesn't work because the remote agent does not install two registry keys Computer\HKEY_CURRENT_USER\Software\Google\Chrome\Extensions\akaagfbdekcffpppnbajieljleihifdc and Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Google\Chrome\Extensions\akaagfbdekcffpppnbajieljleihifdc . AA supports asked customer to install Automation Anywhere also con Citrix Server but you can do it smarter, simply creating those registry keys manually on the server

Some of the things i've written in this little blog post are now in the online documentation, but since it required lot of time before discovering it, if you're on this page, maybe could be useful.

Read More

Authorization using IIS UrlRewrite and headers

Lately one of my customers requested a very simple web application, with a on-premise frontend and a backend which calls Google AI APIs.

Why a on-prem web application? The client leverages on Citrix NetScalar enable authentication through SSO.

What about authorization? They create AD groups and restrisct permission using NetScalar also as authorization layer.

The problem is that, when we were deploying in production, client policies were also changing, so the only way to implement authorization became to read headers (see the example below) and use saml-group header value on the backend.

Unfortunally we built a AngulaJS application, so reading the headers, implement httpModules or other stuff like these was not so easy, so we looked for a cheap solution to mantain the as is solution.

Using just IIS Url Rewrite Module we are able to read headers and filter access according to content without impact on the frontend application or writing a single line of code.

As you can see in this image, with a rule like this, IIS blocks every kind of request that is not coming from a user belonging to group BPO-997_CA...swich.

When referring to a particular header key (in my case saml-groups), you have to use HTTP_saml-groups input in order to create the rule.

What's wrong with this solution? It's applicable only when you need a very simple authorization (in our case only access the application) but, useful to know ;)

Credits to Valerio Angelini and Marco Aimi who worked with me testing this solution.

Read More

pdf2data template editor setup that works

pdf2data is a great product, but the web app install is not so well documented, moreover iText has a terrible customer service that simply doesn't reply to your email.
So if you have to start this web app you simply have this link which does not documents all the passages.
I assume you installed tomcat and following the guide you continue having errors, so:

1. Undeploy your pdf2data web app
2. Rename the downloaded war file in directory "c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\" simply in "pdf2data.war"
3. Create a TEMP folder (eg. C:\TEMP) to host pdf2data temp files
4. Create the file web.properties in c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\pdf2data". dir.temp is not documented and requires double backslash in the path, licensekey must be the absolute path of your itext license key

dir.temp = C:\\TEMP
licensekey = C:\itextkeylicense.xml

mail.to=
mail.smtp.host=
mail.smtp.port=
mail.ssl.smtp.port=
mail.ssl.enable=
mail.smtp.starttls.enable=

5.  Create a system variable PDF2DATA_PROPERTIES and set it to the path of the file from previous step (eg: "c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\pdf2data\web.properties")
6. Restart tomcat and enjoy your pdf2data template editor

Read More