Using the Power BI API with PowerShell scripts – Refreshing your dataset
I know when I started trying to use the Power BI APIs, I was unsure of what I was doing, and how I could get it working.
Not only that, but I could see that as with building any cloud platform the APIs have to be there before they can put the skin on and make it easy to use.
There is always new features or functionality that is available via the Power BI APIs.
In this series of Blog posts, I am going to show how you can use additional features in Power BI by using the API and PowerShell.
Let me start by saying that I am no coding or scripting expert.
My goal in this blog post, is to show that using the PowerShell scripts can be fairly easy, once someone (me, choose ME) has shown you how to put in the required details.
Example
As always, I like to use an example, which I find makes it easier to follow.
In this example, I am going to be using the Power BI Refresh API to refresh a dataset using PowerShell.
My dataset will be “Fourmoo Google Analytics”
Getting the details required for the PowerShell Script
I got the details on how to refresh a Power BI Dataset from the following location below. Here it explains what the Power BI API expects to receive in order to refresh the dataset.
https://msdn.microsoft.com/en-us/library/mt784652.aspx
I then downloaded the PowerShell Script from here: Power BI Dataset Refresh API Sample Code
Would you believe that there are only 3 pieces of information required in order to update the PowerShell Script and get it running?
Below I will explain how to get this data and put it onto the PowerShell Script
Registering for a ClientID
The first step I had to do was to register for a ClientID
- To do this I went to the following URL:
- https://dev.powerbi.com/apps
- If I was not logged in I would be prompted to log in with my Power BI Account.
-
I put in the following below in order to get the ClientID that I need in my PowerShell script.
- I then clicked on Register App
-
Once that completes I then got my ClientID
- It will be displayed in the ClientID section below, which I then copied and pasted and stored in a secure location.
GroupID
NOTE: Group is what it was called in Power BI before it was changed to App Workspaces. So when anything refers to Group, it is actually now referring to an App Workspace.
This is the Group where your dataset currently is in.
To find the GroupID you can complete the following steps below.
- Log into the Power BI Service, and then I went to the App Workspace where I have the dataset that I want to refresh.
-
In my example I went into the “Site Analytics” Workspace
-
Then in the URL you will see that it will say groups, and the section immediately after that is the GroupID
- As you can see above my GroupID starts with 0084
- Make a note of the GroupID
DatasetID
Next is where I needed to find the Dataset ID, which I show in the steps below.
- I made sure I was already in my Site Analytics App Workspace.
-
Then I clicked on Settings and then Settings and then Settings again
-
I then clicked on Datasets, and then clicked on the dataset that I want to refresh.
- As in my example it was called “Fourmoo Google Analytics”
-
Now in the URL I now had my DatasetID which is which is after datasets, and highlighted as shown below.
- I then made a note of the DatasetID
Updating the PowerShell Script
Now I am ready to update the PowerShell script with the details I got from above.
NOTE: The PowerShell script does have a lot of documentation in it, so if you get lost there is a lot of comments to assist you.
- I opened the PowerShell ISE
- I then opened my PowerShell script.
-
Now where it says $groupID I put in my GroupID that I had noted earlier.
-
Then where it said $datasetID, I put in my DatasetID that I had noted earlier.
-
The final piece is to put in the $clientID, which I had noted earlier
- Then I saved the PowerShell Script file
Testing the PowerShell script and my dataset refresh
The final step is to now run the PowerShell script, ensure that it runs, and my dataset actually gets refreshed.
-
As you can see my last refresh is shown below
- I then run the PowerShell Script.
-
I got a prompt asking for the App Permissions
- I clicked Accept
- Then I got the following result showing it was successfully executed
-
I could now see that it was In progress
-
And finally, it was successfully refreshed.
Conclusion
Whilst this has been a bit of a longer blog post, I do hope that you can see that if you follow the steps how you can leverage the Power BI APIs for your datasets.
I do plan on doing additional blog posts on the different Power BI APIs. If there are any that you are interested in, please leave a comment below.
And as I say in every blog post, please leave any comments or questions.
Link to the PowerShell Script here: ManageRefresh.ps1
[…] you need a recap here is a link to my previous blog post Using the Power BI API with PowerShell scripts, in which I demonstrated how to refresh your Power BI dataset using PowerShell and the Power BI […]
I was actually looking for information on Power BI REST API & Powershell. Glad I found yours. I believe mentioning the fact you’re no powershell wizard is a very good introduction. I never pretended to be one either. So we’ll settle with proficient. I also shared same feelings when I first decided to explore Power BI REST API to find out what it was all about. Maybe I’ve missed something, somewhere but there seemed to be a bit of confusion understanding – as well as clearly defining – what Power BI REST API was all about in first place. Maybe the first instinctive relation we could make was mainly embedding scenarios. On June 2017, Microsoft claimed they would offer a ‘fusion’ of previous Azure Portal – Power BI Embedded option with Power BI REST API into one consolidated API. I’d like to add my 2 cents in saying they simply decided to remove what was formerly known as Workspace Collection topology from the map. The need to have a Premium also raised a big debate but I’m not getting into that… The deeper I went, the more I realized Power BI REST API offered a lot more than just embedding options. So I guess as usual… it was mainly up to us to find out what Power BI REST API and Powershell combination can really perform in real context through personal initiative R&D and testing. I do have respect for Microsoft Power BI team though I would have loved seing more consolidated information on Power BI REST API with Powershell. Not having to parse a little bit here and a little bit there…Well you know what I mean. Anyway thanks for sharing and keep on your good work. It’s appreciated.😉👍
Thanks for the comments and you have some great insights into how it could or should work.
From my side, the way I see it is that they have to build the APIs in order to get Power BI Working. Now it appears that they are starting to expose a lot more of the APIs for Power BI, as well as giving PowerShell scripts to use the APIs. And in my opinion this then makes it easier to do thing pragmatically using PowerShell.
Greate articel, but you missed the most important part. “ensure that it runs”.
Thanks for the comment, and you are correct, always ensure that it runs!
what dose “ensure that it runs” means?
I have followed all the instructions provided by you. And it have successfully run and got the result as shown in screeshot of powershell. But I did not find any record in refresh history of dataset.
Please help me go through this.
Thanks.
Hi there,
It appears that it is running because it completes with no errors.
The recordset is returned as JSON so you would need to convert it to view the data
Hi – great walkthrough Gilbert, made it very simple to follow!
I’m getting the same result as Nagendra however. I get the messages to say it’s completed but no history of it in Power BI…?
Hi – I’ve completed the run but I’m getting the message below which differs slightly to what you have:
VERBOSE: GET https://api.powerbi.com/v1.0/myorg/datasets/%5Bdatasetid%5D/refreshes with 0-byte payload
VERBOSE: received -1-byte response of content type application/json; odata.metadata=minimal; odata.streaming=true
There is no history of the refresh showing in the dataset
Thanks,
Rob
Hi Rob,
Have you been successful in refreshing the dataset manually?
Hi Gilbert,
Yes, I’m able to manually refresh. It’s on a personal gateway if that makes any difference?
Thanks,
Rob
Hola he seguido el tutorial a mi también me sale la misma novedad, lo resolviste a la final?
I also go this message once while trying to implement this. If I remember well, you have to refresh the dataset manually (or with a schedule) at least once before you can call this GET. Otherwise, the refresh history is empty, which is why you get a 0-byte payload (No data to show!).
This went great up until the end, when I received a “Sorry, but we’re having trouble signing you in AADSTS50011: The reply url specified in the request does not match the reply urls configured for the application: ” error.
My assumption is that the URL I used in when I registered for the Client ID was incorrect. How do I go about getting the correct Redirect URL for my Power Bi App?
Thanks,
Duncan
Hi Gilbert,
This is a great, step-by-step article and I appreciate how you detail how to get all the necessary ID’s. The one step where I failed seems to be the Redirect URL.
AADSTS50011: The reply url specified in the request does not match the reply urls configured for the application:
Could you please instruct me on how to find the appropriate redirect URL for the Power BI app?
Thanks!
Duncan
Hi Gilbert,
That is an excellent article and hopefully it will resolve an issue I have been having. Every step worked except – apparently – the step where I paste the redirect URL into the Client ID request as I got the following error when I ran the updated script.
AADSTS50011: The reply url specified in the request does not match the reply urls configured for the application:
Could you please instruct me as to how to get the appropriate redirect URL for my Power BI app?
Thanks,
Duncan
Hi there,
I would just create another ClientId with the right redirect URL and try again.
Hi Gilbert,
I seem to be getting the same response as Rob.
VERBOSE: GET https://api.powerbi.com/v1.0/myorg/groups/d75b2a8d-f68f-4659-a88b-5c7829b7b333/datasets/d7ba622f-a2be-4f63-bb52-4cb508dfddeb/refreshes with 0-byte payload
VERBOSE: received -1-byte response of content type application/json; odata.metadata=minimal; odata.streaming=true
The dataset is one that I have successfully refreshed manually many times.
Thanks,
Duncan
Hi Gilbert,
Great article, but I seem to be having the same problem as Rob reported.
The Power BI interface does not reflect that report is being refreshed.
I tried with two different datasets – both of which I have been able to refresh manually many times.
Thanks,
Duncan
Hi Duncan,
I’m not sure why it’s not working. I will test my own tenant later and let you know the outcome.
I think that is it. And why it isn’t working.
I’m certain that I recall that it only works through the enterprise gateway
Hi Gilbert,
I want to assume you are referring to a personal gateway, but you know what people say about assuming things.
I did a little research and couldn’t find an official definition of what an enterprise gateway is; Microsoft says there is a gateway in personal mode and a gateway. Either way, I am not using a personal gateway.
Could you please elaborate?
Thanks,
Duncan
Hi Duncan,
What they mean by Personal and Gateway is that when it is installed it is in the Gateway (Enterprise Mode)
I even tried to refresh a dataset now where I am running it through the Gateway and it is working perfectly fine.
One question is the gateway running the latest version?
Latest version, updated last week.
Hi Gilbert, great article – thanks. I have refreshes working great for me in a manually run PowerShell script, but I want to run it as part of a scheduled task to refresh my API backed dataset more often than the maximum of 8(?) times daily with a cached dataset (ie “manually” but using this script). However, given the script pops up an interactive GUI login box to authenticate, this doesn’t work in a scheduled task. How can I automate the retrieval of credentials – perhaps from an on-disk encrypted file or something?
Hi there
You can certainly script this with an Azure Credential which will run without being prompted to log in.
Just remember that if you are using Power BI Pro you will only get 8 refreshes a day, be that manually or via the PowerShell script.
Hi Gilbert,
This is one of the best articles i have come across while doing data refresh activity for my report. Thank you so much.
I have one query regarding when you say ‘Just remember that if you are using Power BI Pro you will only get 8 refreshes a day, be that manually or via the PowerShell script’.
Standard licensing page of Power refers the refresh of 8 (Pro) and 48 (Premium) as “Maximum number of automatic refreshes per day”. So is there some misinterpretation or I am misunderstanding the terms. Please could you clarify.
Thanks.
Hi there
Thanks for the lovely compliment.
With regards to your question,
If you have a Power BI Pro license, you can refresh your dataset 8 times per day.
If you have a dataset that is hosted on Power BI Premium you can refresh 48 times per day (I think possibly it could be even higher)
Does that answer your question?
Thanks for the answer, however i am able to refresh more than 8 times manually wit Pro License.
I assume there is a catch when we talk about refresh. Do you assume the same. Lets continue our talk on LinkedIn.
Cheers!!
Anubhav
Hi ,
I ‘m unable to download the Managerefresh.ps1 file it’s throwing error. Could you please take a look?
Hi there
I just tried the link and it does work for me.
Possibly it might be being blocked from where you are connecting from?
You had mentioned “I then downloaded the PowerShell Script from here: HTTP”, but where is the link?
Hi there
Thanks for letting me know that I missed the link.
It now has been updated with the link to the PowerShell script.
Yes, it works. Some changes related to “AzureRM.profile” path and some other installations that needs to be done related to “AzureRM” which was not there at my end. Its running successfully right now but the data-set doesn’t seems to be refreshed as per your snapshots that you have shown as an example. Would could be the reason? Any troubleshooting steps if you can share would be grateful to you.
Hi there,
If you go into the dataset settings, can you see it saying that the data refresh is in progress?
Also what is the source of your data?
You are not using DirectQuery or LiveConnection by some chance?
No, we are not using “Direct Query” or “Live Connection”. We are using “Import” data from SQL Server database residing in Azure VM. i have found the reason why the “ManageRefresh.ps1” power shell script was nor refreshing the dataset, it’s because of the “#” remark tag on “Refresh the dataset”. Now the other problem comes, while looking at the report, the report is not refresh through the updated dataset from the PowerShell script. Am i missing anything?
Hi Adil,
I am also using the One Drive file of Manage Refresh. However I am not able to see any record in Refresh History and neither my Dataset is refreshing. What might be the reason for this?
Hi Gilbert,
I have executed the article step by step by using the One Drive file.However there is no record created for me in “Refresh History” of DataSet.
Any possible suggestions?
Hey Gilbert,
Thanks for such a wonderful article.However i do not understand the intention of invoking Refresh Data Set PowerBI API when when it is limited to 8 calls only in a day.Same activity can be performed for a Data set using “Schedule Refresh” timeslots.
Thanks for the script but how to enable the failure email notification with this code. i want whenever the data set gets failed it should send an email to respective people via script.
please advice.
Hi there
You might have to change the way it works if you want it to send when it fails?
Yes, I want to send a failure email to users and owner if data refresh gets failed then, any suggestion would be great help here.
Can you please provide the solution of my request , it is very much needed to business.
[…] https://www.fourmoo.com/2018/06/05/using-the-power-bi-api-with-PowerShell-scripts-refreshing-your-da… […]
Is there a way via powershell script to switch on the schedule of refresh of any dataset?
Hi Bhavya,
I think that this might be possible, you could look at the Power BI REST APIs
Or you could use Power Automate to refresh the datasets based on a specific schedule or action?
looks like there are some changes now in dev.powerbi.com when you register an app. if you choose native app to provide the redirect url. has anyone tested this if still works?
Hi there
If I recall off the top of my head you can put in almost any redirect URL?
Can anyone register on register for ClientID on https://dev.powerbi.com/apps or specific in organization are allowed to register like who have tenant access ?
Hi Rohit,
Your organization can certainly lock it down. I would chat to your network/Azure admin.
RE: tenant access lock down… Yes indeed… this can happen even with no warning… that is something I had to deal with on a mandate for big banking financial institution. It had 1st been clearly discussed with head administration that logically, data analytics directors and managers and team should be granted access to PBI tenants content in order to be able to operate within a data analytics collaboration context in their own given rights. We had 1st brought the creation of Security Groups on AAD with Servive Principal & Azure Key Vault, so they wouldn’t have to be slaved by Office 365 managers whenever came the time to manage tenants content inventory and activities. Everything worked fine… till some Office 365 admins decided otherwise… pulling the switch.., and locking access to tenant admins with no given logical reason.
They just didn’t care.., one less problem to manage as they say… Bang!!!. So the battle began… Most data analytics & PBI Embedded applications + a PowerShell automated tenant inventory and activity analysis project using Power BI REST API had to be put on pause. We then had no other choice than to ask for meeting with head administration with number$… here’s how much you’re wasting vs how much you could gain in productivity, Data analytics team MUST be granted access otherwise you can say adios to everything you invested. A new appointed manager listened to us and as a result tenant administrators were given back access the following days. (YEES). PBI collaboration infrastructure and data analytics applications were completed and whole project is now running and gained a huge succe$$. As for the Office 365 admins… Weeeell… I was told they no longer work for the banking institution… So the choice is clear… either you redundantly give fake title and full power to incompetents who will do everything in their power to avoid any upcoming result or invest in experts who will bring you all the assets; expertise, constant will and energy to make it happen. 😉
Thanks for sharing that is good to know that you now have got access to the data you require. It is essential to be able to report on it!
Though i didn’t get much, Precisely i am capacity admin so should i request to tenant or can i go-ahead and register on https://dev.powerbi.com/apps for ClientID?
Ok, so do you want me to provide you any contact or you able to identify Azure admin yourself ?
Does using a script effectively allow you to skirt Microsoft’s 8 / 48 (Pro/Premium) a day automated refresh limit?
Hi Matt, no it does not the refresh limit still applies!