How to compute Row Size in Complex Spark DataFrame?

-



Definitely there may be a better way of doing this. I was running into an error where data load from lake to CosmosDB is failing due to a record size exceeding 2MB limit. Hence, converting it to JSON grouped by primary key and computing the size seemed appropriate and easy. Comment below if this can be done better. 

Comments

Post a Comment

Feedback - positive or negative is welcome.

Popular posts from this blog

Updating SourceData/ Data Source of the Pivot Table

-
Image
Errors:  1. Run time error '1004' Cannot Open PivotTable source file https://....[servlet.FileDownload?]... 2. Data source reference is not valid 3. Can not use web data source as pivot data 4.  Run-time error '-2147024809 (80070057)' Item with specified name wasn't found 5.  Error: Cannot open PivotTable source file ‘[filename[x].xls]SourceData’ 6.  Exception from HRESULT: 0x800A03EC   at System.Dynamic.ComRuntimeHelpers.CheckThrowException(Int32 hresult, ExcepInfo& excepInfo, UInt32 argErr, String message)    at CallSite.Target(Closure , CallSite , ComObject , String )    at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)    at CallSite.Target(Closure , CallSite , Object , String )    at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1) Solution: We have spend a day searching around for the solution. We ...
Read more

Salesforce.com migration tool - Deploying Weblink and migrating files with special characters

-
Image
Recently, I noticed the followed issues in SFDC Ant deployment. Issue 1: Typo in XML <name> metadata data for Web Link Just to reiterate, all the buttons and links created under "Buttons and Links" section of the Object are treated as Web Links . Screenshot from Force.com Migration Tool Guide It should be WebLink, not Weblink. Otherwise, the following error occurs while extracting the package :  package.xml - Entity type: 'Weblink' is unknown. Correct XML is:  <types> <members>Opportunity.Cancel_CheckOut</members> <members>Opportunity.Generate_Deal_Sheet</members> <name>WebLink</name> </types> Issue 2: Unable to deploy entities with special characters Example: <types> <members> User w/Export  </members> <name>Profile</name> </types> I tried to deploy Profile entities with special characters like '-' and '/...
Read more

Information Architecture - Setup your term store to scale

-
In this post I am going to put together a list of best practices to consider when setting up a term store in Sharepoint Online. As you may already know term store is collection of term groups. And each term group consists of number term sets. Each term set contains a list/hierarchy of terms. Team group gives the flexibility in terms of access. You can assign permissions to users on their term group, rather than giving them access to the entire store.  Always maintain a corporate/general term group which contains common term sets across organization.  Every group/business unit should have a term group. Pin the terms from corporate term group as required into Business unit term groups.  For example, you plan to maintain list of document types in common term group. Thus, there is a term set created. Marketing department want to use this term set. But, they want to add a few more terms. In this case, you should create another term set in Marketing department term g...
Read more