How to compute Row Size in Complex Spark DataFrame?

-



Definitely there may be a better way of doing this. I was running into an error where data load from lake to CosmosDB is failing due to a record size exceeding 2MB limit. Hence, converting it to JSON grouped by primary key and computing the size seemed appropriate and easy. Comment below if this can be done better. 

Comments

Post a Comment

Feedback - positive or negative is welcome.

Popular posts from this blog

How to prepare your LOB app for Intune?

-
Image
One of the aspects I struggled with Intune is to wrap an LOB app. In the first cut, it feels like just wrapping the app is enough. But, you need to understand some subtle architectural aspects to get your wrapping right.  App features To get started, let's start with app features.  App uses Azure AD authentication App is a hybrid and uses Cordova (Supports iOS and Android) App communicates with Sharpoint online (via REST) and set up as a Azure AD native application. Thus, conditional access is involved here.  "Send logs via email" - this feature is important as wrapping the application will restrict the email/data sharing capabilities based on the policy  Intune Way You can wrap the application in two ways Using Intune Wrapping Tool Use the Cordova plugin Please visit the link for more details  https://docs.microsoft.com/en-us/intune/deploy-use/decide-how-to-prepare-apps-for-mobile-application-management-with-microsoft-intune Intune wrappi
Read more

Information Architecture - Setup your term store to scale

-
In this post I am going to put together a list of best practices to consider when setting up a term store in Sharepoint Online. As you may already know term store is collection of term groups. And each term group consists of number term sets. Each term set contains a list/hierarchy of terms. Team group gives the flexibility in terms of access. You can assign permissions to users on their term group, rather than giving them access to the entire store.  Always maintain a corporate/general term group which contains common term sets across organization.  Every group/business unit should have a term group. Pin the terms from corporate term group as required into Business unit term groups.  For example, you plan to maintain list of document types in common term group. Thus, there is a term set created. Marketing department want to use this term set. But, they want to add a few more terms. In this case, you should create another term set in Marketing department term group and pin the te
Read more

Generate token signing .CER from ADFS Federation Metadata XML

-
Image
While workging on Force.com SSO, our ADFS team has provided me with federation metadata xml only. As per this link , you also need a token-signing certificate from provider to complete the setup and provide the Force.com XML file to ADFS. Now, the question is how to generate .pem/.cer file out of FederationMetadata.xml file.  Edit FederationMetadata.xml file, and search for  <KeyDescriptor use="signing">. You should find more than one entry. Pick any one of them.      2.  Pick the X500Certificate value and save the text as .der file     3.  openssl x509 -in <(base64 --decode FILE_FROM_STEP2.der) -inform DER -out OUTPUT.pem Use <OUTPUT>.per as Identity Provider Certificate.  Reference:  https://ask.auth0.com/t/how-to-convert-saml-federationmetadata-xml-key-to-pem-or-cer/24
Read more