How to compute Row Size in Complex Spark DataFrame?

-



Definitely there may be a better way of doing this. I was running into an error where data load from lake to CosmosDB is failing due to a record size exceeding 2MB limit. Hence, converting it to JSON grouped by primary key and computing the size seemed appropriate and easy. Comment below if this can be done better. 

Comments

Post a Comment

Feedback - positive or negative is welcome.

Popular posts from this blog

COM Add-in Deployment Issues

-
Environment:  Visual Studio 2010, Office 2010, Excel 2010 Error: "Another version of this product is already installed"? Solution: 1.       Try uninstalling the application from "Add and remove programs" 2.       If the above step doesn't resolve the issue, run EXE from the command prompt with /x option 3.       Delete Application cache On Vista, it is located at C:\Users\ \AppData\Local\Apps\2.0 4.       As a last step take a backup of the registry. Search for "DSExcelAdd-in" and delete the corresponding entries. PLEASE TAKE THE BACKUP as needed. (esp. Keys under VSTA and VSTO (HKEY_USERS) Error: Name: From: file:///C:/publish/DSExcelAdd-in.vsto ************** Exception Text ************** System.ArgumentException: Value does not fall within the expected range. at Microsoft.VisualStudio.Tools.Applications.Deployment...
Read more

Salesforce.com migration tool - Deploying Weblink and migrating files with special characters

-
Image
Recently, I noticed the followed issues in SFDC Ant deployment. Issue 1: Typo in XML <name> metadata data for Web Link Just to reiterate, all the buttons and links created under "Buttons and Links" section of the Object are treated as Web Links . Screenshot from Force.com Migration Tool Guide It should be WebLink, not Weblink. Otherwise, the following error occurs while extracting the package :  package.xml - Entity type: 'Weblink' is unknown. Correct XML is:  <types> <members>Opportunity.Cancel_CheckOut</members> <members>Opportunity.Generate_Deal_Sheet</members> <name>WebLink</name> </types> Issue 2: Unable to deploy entities with special characters Example: <types> <members> User w/Export  </members> <name>Profile</name> </types> I tried to deploy Profile entities with special characters like '-' and '/...
Read more

Sample code to upload a file to chatter feed using REST API

-
Image
Step 1: Download required files Download the following files from internet httpcomponents-client-4.2.3-bin.zip org.apache.commons.httpclient.jar From the files, I have added following JARS to my project in eclipse.    Screenshot from Ecclipse Step 2: Get Access Token Through Chrome Login to your Salesforce instance and navigate to Setup -> Create -> Apps.  Create a custom app and fill Client ID and CallBack URL (ex: http://www.google.com) in the following URL. Encode the CallBack URL before passing it as a parameter.  https:// <instance_name> .salesforce.com/services/oauth2/authorize?response_type=token&client_id= <Client_ID_Goes_Here> &redirect_uri= <CallBack_URL> Paste this URL in chrome In Developer Tool -> Network select "preserve log upon navigation" Load the page. We see below page. The page will take us to RemoteAccessAuthorizationPage.apexp which inturn will lea...
Read more