Send Datagrams to an Azure Service Bus with PowerShell and visualize it with Power BI

Marcel Meurer's picture

Service Buses / Event Hubs in Azure are very useful to collect huge amounts of short data telegrams, queue them, process them with Azure Stream Analytics and output them. The output queue can be an SQL-Server, Storage-Blob, Power BI, etc.

I wrote a PowerShell Module to send short datagrams with PowerShell to an Event Hub. In my example I send the logon times of windows clients and visualize these times on a world map with Power BI.

Prepare the Service Bus / Event Hub (Input queue)

Go to the legacy Azure Portal at https://manage.windowsazure.com to create the Event Hub:

Configure access for the windows clients to write datagrams to the Event Hub. Make sure to remember the policy name “WriteDataGram” and the primary key. You need them later to send the data with PowerShell.

Note the shared access key “RootManageSharedAccessKey” for the Event Hub. The policy and key are needed to configure the Stream Analytic job.

Build the Stream Analytic Job

Stream Analytic receives data from the Event Hub, computes the data and forwards this to an output queue (in this case: Power BI). To create the job go to the Azure Portal at https://portal.azure.com

Add the input queue (the Event Hub) and configure it:

Add an output queue:  

To add Power BI (preview) you have to configure it in the legacy portal (https://manage.windowsazure.com):

Authorize to PowerBI:

Configure how the data is visible in Power BI:

Configure the Stream Analytics query. For testing, I will send all input datagrams to Power BI without processing (later I will show the logon times on a world map in a tumbling window – so the map only shows the logons of the last hour).

Start the Stream Analytic job.

Send datagrams with PowerShell

To easily send data, you can install the Azure.EventHub module from https://www.powershellgallery.com/packages/Azure.EventHub with:

Install-Module -Name Azure.EventHub -Scope CurrentUser

#To send a datagram:
$DataGram='{ "DeviceId":"0002222aa2eeffa", "City":"London, UK","LogonDurationInSec":"13"}'

$URI="ehworkplaceclients-ns.servicebus.windows.net/ehworkplaceclients"
$SASToken=Get-AzureEHSASToken -URI $URI -AccessPolicyName "WriteDataGram" -AccessPolicyKey "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/IYydJOGNiU="

Send-AzureEHDatagram -URI $URI -SASToken $SASToken -Datagram $DataGram

Response:

StatusCode        : 201
StatusDescription : Created
Content           :
RawContent        : HTTP/1.1 201 Created
                    Transfer-Encoding: chunked
                    Content-Type: application/xml; charset=utf-8
                    Date: Thu, 24 Mar 2016 08:41:42 GMT
                    Server: Microsoft-HTTPAPI/2.0


Forms             : {}
Headers           : {[Transfer-Encoding, chunked], [Content-Type, application/xml; charset=utf-8], [Date, Thu, 24 Mar
                    2016 08:41:42 GMT], [Server, Microsoft-HTTPAPI/2.0]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 0

Power BI

There is a new dataset in Power BI, created by the Stream Analytic job. This data can be used in tables, charts, maps, etc. Here is the test datagram displayed:u  

Add new comment
By submitting this form, you accept the Mollom privacy policy.